././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1745324802.7136004
lxml-5.4.0/ 0000755 0001751 0000200 00000000000 15001705403 012057 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/CHANGES.txt 0000644 0001751 0000200 00000406156 15001705132 013703 0 ustar 00runner docker ==============
lxml changelog
==============
5.4.0 (2025-04-22)
==================
Bugs fixed
----------
* Binary wheels use libxml2 2.13.8 and libxslt 1.1.43 to resolve several CVEs.
(Binary wheels for Windows continue to use a patched libxml2 2.11.9 and libxslt 1.1.39.)
5.3.2 (2025-04-05)
==================
This release resolves CVE-2025-24928 as described in
https://gitlab.gnome.org/GNOME/libxml2/-/issues/847
Bugs fixed
----------
* Binary wheels use libxml2 2.12.10 and libxslt 1.1.42.
* Binary wheels for Windows use a patched libxml2 2.11.9 and libxslt 1.1.39.
5.3.1 (2025-02-09)
==================
Bugs fixed
----------
* GH#440: Some tests were adapted for libxml2 2.14.0.
Patch by Nick Wellnhofer.
* LP#2097175: ``DTD(external_id="…")`` erroneously required a byte string as ID value.
* GH#450: ``iterparse()`` internally triggered the `DeprecationWarning`` added in lxml 5.3.0 when parsing HTML.
Other changes
-------------
* GH#442: Binary wheels for macOS no longer use the linker flag ``-flat_namespace``.
5.3.0 (2024-08-10)
==================
Features added
--------------
* GH#421: Nested ``CDATA`` sections are no longer rejected but split on output
to represent ``]]>`` correctly.
Patch by Gertjan Klein.
Bugs fixed
----------
* LP#2060160: Attribute values serialised differently in ``xmlfile.element()`` and ``xmlfile.write()``.
* LP#2058177: The ISO-Schematron implementation could fail on unknown prefixes.
Patch by David Lakin.
Other changes
-------------
* LP#2067707: The ``strip_cdata`` option in ``HTMLParser()`` turned out to be useless and is now deprecated.
* Binary wheels use the library versions libxml2 2.12.9 and libxslt 1.1.42.
* Windows binary wheels use the library versions libxml2 2.11.8 and libxslt 1.1.39.
* Built with Cython 3.0.11.
5.2.2 (2024-05-12)
==================
Bugs fixed
----------
* GH#417: The ``test_feed_parser`` test could fail if ``lxml_html_clean`` was not installed.
It is now skipped in that case.
* LP#2059910: The minimum CPU architecture for the Linux x86 binary wheels was set back to
"core2", without SSE 4.2.
* If libxml2 uses iconv, the compile time version is available as `etree.ICONV_COMPILED_VERSION`.
5.2.1 (2024-04-02)
==================
Bugs fixed
----------
* LP#2059910: The minimum CPU architecture for the Linux x86 binary wheels was set back to
"core2", but with SSE 4.2 enabled.
* LP#2059977: ``Element.iterfind("//absolute_path")`` failed with a ``SyntaxError``
where it should have issued a warning.
* GH#416: The documentation build was using the non-standard ``which`` command.
Patch by Michał Górny.
5.2.0 (2024-03-30)
==================
Other changes
-------------
* LP#1958539: The ``lxml.html.clean`` implementation suffered from several (only if used)
security issues in the past and was now extracted into a separate library:
https://github.com/fedora-python/lxml_html_clean
Projects that use lxml without "lxml.html.clean" will not notice any difference,
except that they won't have potentially vulnerable code installed.
The module is available as an "extra" setuptools dependency "lxml[html_clean]",
so that Projects that need "lxml.html.clean" will need to switch their requirements
from "lxml" to "lxml[html_clean]", or install the new library themselves.
* The minimum CPU architecture for the Linux x86 binary wheels was upgraded to
"sandybridge" (launched 2011), and glibc 2.28 / gcc 12 (manylinux_2_28) wheels were added.
* Built with Cython 3.0.10.
5.1.2 (2024-??-??)
==================
Bugs fixed
----------
* LP#2059977: ``Element.iterfind("//absolute_path")`` failed with a ``SyntaxError``
where it should have issued a warning.
5.1.1 (2024-03-28)
==================
Bugs fixed
----------
* LP#2048920: ``iterlinks()`` in ``lxml.html`` rejected ``bytes`` input in 5.1.0.
* High source line numbers from the parser are no longer truncated
(up to a C ``long``) when using libxml2 2.11 or later.
Other changes
-------------
* GH#407: A compatibility test was adapted to recent expat versions.
Patch by Miro Hrončok.
* Binary wheels use the library versions libxml2 2.12.6 and libxslt 1.1.39.
* Windows binary wheels use the library versions libxml2 2.11.7 and libxslt 1.1.39.
* Built with Cython 3.0.9.
5.1.0 (2024-01-05)
==================
Features added
--------------
* Parsing ASCII strings is slightly faster.
Bugs fixed
----------
* GH#349: The HTML ``Cleaner()`` interpreted an accidentally provided string parameter
for the ``host_whitelist`` as list of characters and silently failed to reject any hosts.
Passing a non-collection is now rejected.
Other changes
-------------
* Support for Python 2.7 and Python versions < 3.6 was removed.
* The wheel build was migrated to use ``cibuildwheel``.
Patch by Primož Godec.
5.0.2 (2024-03-28)
==================
Other changes
-------------
* GH#407: A compatibility test was adapted to recent expat versions.
Patch by Miro Hrončok.
* Binary wheels use the library versions libxml2 2.12.6 and libxslt 1.1.39.
* Built with Cython 3.0.9.
5.0.1 (2024-01-05)
==================
Bugs fixed
----------
* LP#2046208: Parsing non-BMP Python Unicode strings could fail on macOS.
* LP#2044225: When incrementally parsing broken HTML, reporting start events on
missing structural tags failed and could lead to subsequent exceptions.
* LP#2045435: Some (not all) issues with stricter C compilers were resolved.
* The binary wheels in the 5.0.0 release did not validate cleanly (but installed ok).
.. _latest_release:
5.0.0 (2023-12-29)
==================
Features added
--------------
* Character escaping in ``C14N2`` serialisation now uses a single pass over the text
instead of searching for each unescaped character separately.
* Early support for Python 3.13a2 was added.
Bugs fixed
----------
* LP#1976304: The ``Element.addnext()`` method previously inserted the new element
before existing tail text. The tail text of both sibling elements now stays on
the respective elements.
* LP#1980767, GH#379: ``TreeBuilder.close()`` could fail with a ``TypeError`` after
parsing incorrect input. Original patch by Enrico Minack.
* ``Element.itertext(with_tail=False)`` returned the tail text of comments and
processing instructions, despite the explicit option.
* GH#370: A crash with recent libxml2 2.11.x versions was resolved.
Patch by Michael Schlenker.
* A compile problem with recent libxml2 2.12.x versions was resolved.
* The internal exception handling in C callbacks was improved for Cython 3.0.
* The exception declarations of ``xmlInputReadCallback``, ``xmlInputCloseCallback``,
``xmlOutputWriteCallback`` and ``xmlOutputCloseCallback`` in ``tree.pxd`` were
corrected to prevent running Python code or calling into the C-API with a live
exception set.
* GH#385: The long deprecated ``unittest.m̀akeSuite()`` function is no longer used.
Patch by Miro Hrončok.
* LP#1522052: A file-system specific test is now optional and should no longer fail
on systems that don't support it.
* GH#392: Some tests were adapted for libxml2 2.13.
Patch by Nick Wellnhofer.
* Contains all fixes from lxml 4.9.4.
Other changes
-------------
* LP#1742885: lxml no longer expands external entities (XXE) by default to prevent
the security risk of loading arbitrary files and URLs. If this feature is needed,
it can be enabled in a backwards compatible way by using a parser with the option
``resolve_entities=True``. The new default is ``resolve_entities='internal'``.
* With libxml2 2.10.4 and later (as provided by the lxml 5.0 binary wheels),
parsing HTML tags with "prefixes" no longer builds a namespace dictionary
in ``nsmap`` but considers the ``prefix:name`` string the actual tag name.
With older libxml2 versions, since 2.9.11, the prefix was removed. Before
that, the prefix was parsed as XML prefix.
lxml 5.0 does not try to hide this difference but now changes the ElementPath
implementation to let ``element.find("part1:part2")`` search for the tag
``part1:part2`` in documents parsed as HTML, instead of looking only for ``part2``.
* LP#2024343: The validation of the schema file itself is now optional in the
ISO-Schematron implementation. This was done because some lxml distributions
discard the RNG validation schema file due to licensing issues. The validation
can now always be disabled with ``Schematron(..., validate_schema=False)``.
It is enabled by default if available and disabled otherwise. The module
constant ``lxml.isoschematron.schematron_schema_valid_supported`` can be used
to detect whether schema file validation is available.
* Some redundant and long deprecated methods were removed:
``parser.setElementClassLookup()``,
``xslt_transform.apply()``,
``xpath.evaluate()``.
* Some incorrect declarations were removed from ``python.pxd``. In general, this file
should not be used by external Cython code. Use the C-API declarations provided by
Cython itself instead.
* Binary wheels use the library versions libxml2 2.12.3 and libxslt 1.1.39.
* Built with Cython 3.0.7, updated to follow recent changes in Cython 3.1-dev.
4.9.4 (2023-12-19)
==================
Bugs fixed
----------
* LP#2046398: Inserting/replacing an ancestor into a node's children could loop indefinitely.
* LP#1980767, GH#379: ``TreeBuilder.close()`` could fail with a ``TypeError`` after
parsing incorrect input. Original patch by Enrico Minack.
* LP#1522052: A file-system specific test is now optional and should no longer fail
on systems that don't support it.
Other changes
-------------
* Wheels include zlib 1.3, libxml2 2.10.3 and libxslt 1.1.39
(zlib 1.2.12, libxml2 2.10.3 and libxslt 1.1.37 on Windows).
* Built with Cython 0.29.37.
4.9.3 (2023-07-05)
==================
Bugs fixed
----------
* LP#2008911: ``lxml.objectify`` accepted non-decimal numbers like ``²²²`` as integers.
* A memory leak in ``lxml.html.clean`` was resolved by switching to Cython 0.29.34+.
* GH#348: URL checking in the HTML cleaner was improved.
Patch by Tim McCormack.
* GH#371, GH#373: Some regex strings were changed to raw strings to fix Python warnings.
Patches by Jakub Wilk and Anthony Sottile.
Other changes
-------------
* Wheels include zlib 1.2.13, libxml2 2.10.3 and libxslt 1.1.38
(zlib 1.2.12, libxml2 2.10.3 and libxslt 1.1.37 on Windows).
* Built with Cython 0.29.36 to adapt to changes in Python 3.12.
4.9.2 (2022-12-13)
==================
Bugs fixed
----------
* CVE-2022-2309: A Bug in libxml2 2.9.1[0-4] could let namespace declarations
from a failed parser run leak into later parser runs. This bug was worked around
in lxml and resolved in libxml2 2.10.0.
https://gitlab.gnome.org/GNOME/libxml2/-/issues/378
Other changes
-------------
* LP#1981760: ``Element.attrib`` now registers as ``collections.abc.MutableMapping``.
* lxml now has a static build setup for macOS on ARM64 machines (not used for building wheels).
Patch by Quentin Leffray.
4.9.1 (2022-07-01)
==================
Bugs fixed
----------
* A crash was resolved when using ``iterwalk()`` (or ``canonicalize()``)
after parsing certain incorrect input. Note that ``iterwalk()`` can crash
on *valid* input parsed with the same parser *after* failing to parse the
incorrect input.
4.9.0 (2022-06-01)
==================
Bugs fixed
----------
* GH#341: The mixin inheritance order in ``lxml.html`` was corrected.
Patch by xmo-odoo.
Other changes
-------------
* Built with Cython 0.29.30 to adapt to changes in Python 3.11 and 3.12.
* Wheels include zlib 1.2.12, libxml2 2.9.14 and libxslt 1.1.35
(libxml2 2.9.12+ and libxslt 1.1.34 on Windows).
* GH#343: Windows-AArch64 build support in Visual Studio.
Patch by Steve Dower.
4.8.0 (2022-02-17)
==================
Features added
--------------
* GH#337: Path-like objects are now supported throughout the API instead of just strings.
Patch by Henning Janssen.
* The ``ElementMaker`` now supports ``QName`` values as tags, which always override
the default namespace of the factory.
Bugs fixed
----------
* GH#338: In lxml.objectify, the XSI float annotation "nan" and "inf" were spelled in
lower case, whereas XML Schema datatypes define them as "NaN" and "INF" respectively.
Patch by Tobias Deiminger.
Other changes
-------------
* Built with Cython 0.29.28.
4.7.1 (2021-12-13)
==================
Features added
--------------
* Chunked Unicode string parsing via ``parser.feed()`` now encodes the input data
to the native UTF-8 encoding directly, instead of going through ``Py_UNICODE`` /
``wchar_t`` encoding first, which previously required duplicate recoding in most cases.
Bugs fixed
----------
* The standard namespace prefixes were mishandled during "C14N2" serialisation on Python 3.
See https://mail.python.org/archives/list/lxml@python.org/thread/6ZFBHFOVHOS5GFDOAMPCT6HM5HZPWQ4Q/
* ``lxml.objectify`` previously accepted non-XML numbers with underscores (like "1_000")
as integers or float values in Python 3.6 and later. It now adheres to the number
format of the XML spec again.
* LP#1939031: Static wheels of lxml now contain the header files of zlib and libiconv
(in addition to the already provided headers of libxml2/libxslt/libexslt).
Other changes
-------------
* Wheels include libxml2 2.9.12+ and libxslt 1.1.34 (also on Windows).
4.7.0 (2021-12-13)
==================
* Release retracted due to missing files in lxml/includes/.
4.6.5 (2021-12-12)
==================
Bugs fixed
----------
* A vulnerability (GHSL-2021-1038) in the HTML cleaner allowed sneaking script
content through SVG images (CVE-2021-43818).
* A vulnerability (GHSL-2021-1037) in the HTML cleaner allowed sneaking script
content through CSS imports and other crafted constructs (CVE-2021-43818).
4.6.4 (2021-11-01)
==================
Features added
--------------
* GH#317: A new property ``system_url`` was added to DTD entities.
Patch by Thirdegree.
* GH#314: The ``STATIC_*`` variables in ``setup.py`` can now be passed via env vars.
Patch by Isaac Jurado.
4.6.3 (2021-03-21)
==================
Bugs fixed
----------
* A vulnerability (CVE-2021-28957) was discovered in the HTML Cleaner by Kevin Chung,
which allowed JavaScript to pass through. The cleaner now removes the HTML5
``formaction`` attribute.
4.6.2 (2020-11-26)
==================
Bugs fixed
----------
* A vulnerability (CVE-2020-27783) was discovered in the HTML Cleaner by Yaniv Nizry,
which allowed JavaScript to pass through. The cleaner now removes more sneaky
"style" content.
4.6.1 (2020-10-18)
==================
Bugs fixed
----------
* A vulnerability was discovered in the HTML Cleaner by Yaniv Nizry, which allowed
JavaScript to pass through. The cleaner now removes more sneaky "style" content.
4.6.0 (2020-10-17)
==================
Features added
--------------
* GH#310: ``lxml.html.InputGetter`` supports ``__len__()`` to count the number of input fields.
Patch by Aidan Woolley.
* ``lxml.html.InputGetter`` has a new ``.items()`` method to ease processing all input fields.
* ``lxml.html.InputGetter.keys()`` now returns the field names in document order.
* GH-309: The API documentation is now generated using ``sphinx-apidoc``.
Patch by Chris Mayo.
Bugs fixed
----------
* LP#1869455: C14N 2.0 serialisation failed for unprefixed attributes
when a default namespace was defined.
* ``TreeBuilder.close()`` raised ``AssertionError`` in some error cases where it
should have raised ``XMLSyntaxError``. It now raises a combined exception to
keep up backwards compatibility, while switching to ``XMLSyntaxError`` as an
interface.
4.5.2 (2020-07-09)
==================
Bugs fixed
----------
* ``Cleaner()`` now validates that only known configuration options can be set.
* LP#1882606: ``Cleaner.clean_html()`` discarded comments and PIs regardless of the
corresponding configuration option, if ``remove_unknown_tags`` was set.
* LP#1880251: Instead of globally overwriting the document loader in libxml2, lxml now
sets it per parser run, which improves the interoperability with other users of libxml2
such as libxmlsec.
* LP#1881960: Fix build in CPython 3.10 by using Cython 0.29.21.
* The setup options "--with-xml2-config" and "--with-xslt-config" were accidentally renamed
to "--xml2-config" and "--xslt-config" in 4.5.1 and are now available again.
4.5.1 (2020-05-19)
==================
Bugs fixed
----------
* LP#1570388: Fix failures when serialising documents larger than 2GB in some cases.
* LP#1865141, GH#298: ``QName`` values were not accepted by the ``el.iter()`` method.
Patch by xmo-odoo.
* LP#1863413, GH#297: The build failed to detect libraries on Linux that are only
configured via pkg-config.
Patch by Hugh McMaster.
4.5.0 (2020-01-29)
==================
Features added
--------------
* A new function ``indent()`` was added to insert tail whitespace for pretty-printing
an XML tree.
Bugs fixed
----------
* LP#1857794: Tail text of nodes that get removed from a document using item
deletion disappeared silently instead of sticking with the node that was removed.
Other changes
-------------
* MacOS builds are 64-bit-only by default.
Set CFLAGS and LDFLAGS explicitly to override it.
* Linux/MacOS Binary wheels now use libxml2 2.9.10 and libxslt 1.1.34.
* LP#1840234: The package version number is now available as ``lxml.__version__``.
4.4.3 (2020-01-28)
==================
Bugs fixed
----------
* LP#1844674: ``itertext()`` was missing tail text of comments and PIs since 4.4.0.
4.4.2 (2019-11-25)
==================
Bugs fixed
----------
* LP#1835708: ``ElementInclude`` incorrectly rejected repeated non-recursive
includes as recursive.
Patch by Rainer Hausdorf.
4.4.1 (2019-08-11)
==================
Bugs fixed
----------
* LP#1838252: The order of an OrderedDict was lost in 4.4.0 when passing it as
attrib mapping during element creation.
* LP#1838521: The package metadata now lists the supported Python versions.
4.4.0 (2019-07-27)
==================
Features added
--------------
* ``Element.clear()`` accepts a new keyword argument ``keep_tail=True`` to clear
everything but the tail text. This is helpful in some document-style use cases
and for clearing the current element in ``iterparse()`` and pull parsing.
* When creating attributes or namespaces from a dict in Python 3.6+, lxml now
preserves the original insertion order of that dict, instead of always sorting
the items by name. A similar change was made for ElementTree in CPython 3.8.
See https://bugs.python.org/issue34160
* Integer elements in ``lxml.objectify`` implement the ``__index__()`` special method.
* GH#269: Read-only elements in XSLT were missing the ``nsmap`` property.
Original patch by Jan Pazdziora.
* ElementInclude can now restrict the maximum inclusion depth via a ``max_depth``
argument to prevent content explosion. It is limited to 6 by default.
* The ``target`` object of the XMLParser can have ``start_ns()`` and ``end_ns()``
callback methods to listen to namespace declarations.
* The ``TreeBuilder`` has new arguments ``comment_factory`` and ``pi_factory`` to
pass factories for creating comments and processing instructions, as well as
flag arguments ``insert_comments`` and ``insert_pis`` to discard them from the
tree when set to false.
* A `C14N 2.0 `` configuration.
Bugs fixed
----------
* GH#251: HTML comments were handled incorrectly by the soupparser.
Patch by mozbugbox.
* LP#1654544: The html5parser no longer passes the ``useChardet`` option
if the input is a Unicode string, unless explicitly requested. When parsing
files, the default is to enable it when a URL or file path is passed (because
the file is then opened in binary mode), and to disable it when reading from
a file(-like) object.
Note: This is a backwards incompatible change of the default configuration.
If your code parses byte strings/streams and depends on character detection,
please pass the option ``guess_charset=True`` explicitly, which already worked
in older lxml versions.
* LP#1703810: ``etree.fromstring()`` failed to parse UTF-32 data with BOM.
* LP#1526522: Some RelaxNG errors were not reported in the error log.
* LP#1567526: Empty and plain text input raised a TypeError in soupparser.
* LP#1710429: Uninitialised variable usage in HTML diff.
* LP#1415643: The closing tags context manager in ``xmlfile()`` could continue
to output end tags even after writing failed with an exception.
* LP#1465357: ``xmlfile.write()`` now accepts and ignores None as input argument.
* Compilation under Py3.7-pre failed due to a modified function signature.
Other changes
-------------
* The main module source files were renamed from ``lxml.*.pyx`` to plain
``*.pyx`` (e.g. ``etree.pyx``) to simplify their handling in the build
process. Care was taken to keep the old header files as fallbacks for
code that compiles against the public C-API of lxml, but it might still
be worth validating that third-party code does not notice this change.
3.8.0 (2017-06-03)
==================
Features added
--------------
* ``ElementTree.write()`` has a new option ``doctype`` that writes out a
doctype string before the serialisation, in the same way as ``tostring()``.
* GH#220: ``xmlfile`` allows switching output methods at an element level.
Patch by Burak Arslan.
* LP#1595781, GH#240: added a PyCapsule Python API and C-level API for
passing externally generated libxml2 documents into lxml.
* GH#244: error log entries have a new property ``path`` with an XPath
expression (if known, None otherwise) that points to the tree element
responsible for the error. Patch by Bob Kline.
* The namespace prefix mapping that can be used in ElementPath now injects
a default namespace when passing a None prefix.
Bugs fixed
----------
* GH#238: Character escapes were not hex-encoded in the ``xmlfile`` serialiser.
Patch by matejcik.
* GH#229: fix for externally created XML documents. Patch by Theodore Dubois.
* LP#1665241, GH#228: Form data handling in lxml.html no longer strips the
option values specified in form attributes but only the text values.
Patch by Ashish Kulkarni.
* LP#1551797: revert previous fix for XSLT error logging as it breaks
multi-threaded XSLT processing.
* LP#1673355, GH#233: ``fromstring()`` html5parser failed to parse byte strings.
Other changes
-------------
* The previously undocumented ``docstring`` option in ``ElementTree.write()``
produces a deprecation warning and will eventually be removed.
3.7.4 (2017-??-??)
==================
Bugs fixed
----------
* LP#1551797: revert previous fix for XSLT error logging as it breaks
multi-threaded XSLT processing.
* LP#1673355, GH#233: ``fromstring()`` html5parser failed to parse byte strings.
3.7.3 (2017-02-18)
==================
Bugs fixed
----------
* GH#218 was ineffective in Python 3.
* GH#222: ``lxml.html.submit_form()`` failed in Python 3.
Patch by Jakub Wilk.
3.7.2 (2017-01-08)
==================
* GH#220: ``xmlfile`` allows switching output methods at an element level.
Patch by Burak Arslan.
Bugs fixed
----------
* Work around installation problems in recent Python 2.7 versions
due to FTP download failures.
* GH#219: ``xmlfile.element()`` was not properly quoting attribute values.
Patch by Burak Arslan.
* GH#218: ``xmlfile.element()`` was not properly escaping text content of
script/style tags. Patch by Burak Arslan.
3.7.1 (2016-12-23)
==================
* No source changes, issued only to solve problems with the
binary packages released for 3.7.0.
3.7.0 (2016-12-10)
==================
Features added
--------------
* GH#217: ``XMLSyntaxError`` now behaves more like its ``SyntaxError``
baseclass. Patch by Philipp A.
* GH#216: ``HTMLParser()`` now supports the same ``collect_ids`` parameter
as ``XMLParser()``. Patch by Burak Arslan.
* GH#210: Allow specifying a serialisation method in ``xmlfile.write()``.
Patch by Burak Arslan.
* GH#203: New option ``default_doctype`` in ``HTMLParser`` that allows
disabling the automatic doctype creation. Patch by Shadab Zafar.
* GH#201: Calling the method ``.set('attrname')`` without value argument
(or ``None``) on HTML elements creates an attribute without value that
serialises like ``
``. Patch by Daniel Holth.
* GH#197: Ignore form input fields in ``form_values()`` when they are
marked as ``disabled`` in HTML. Patch by Kristian Klemon.
Bugs fixed
----------
* GH#206: File name and line number were missing from XSLT error messages.
Patch by Marcus Brinkmann.
Other changes
-------------
* Log entries no longer allow anything but plain string objects as message text
and file name.
* ``zlib`` is included in the list of statically built libraries.
3.6.4 (2016-08-20)
==================
* GH#204, LP#1614693: build fix for MacOS-X.
3.6.3 (2016-08-18)
==================
* LP#1614603: change linker flags to build multi-linux wheels
3.6.2 (2016-08-18)
==================
* LP#1614603: release without source changes to provide cleanly built Linux wheels
3.6.1 (2016-07-24)
==================
Features added
--------------
* GH#180: Separate option ``inline_style`` for Cleaner that only removes ``style``
attributes instead of all styles. Patch by Christian Pedersen.
* GH#196: Windows build support for Python 3.5. Contribution by Maximilian Hils.
Bugs fixed
----------
* GH#199: Exclude ``file`` fields from ``FormElement.form_values`` (as browsers do).
Patch by Tomas Divis.
* GH#198, LP#1568167: Try to provide base URL from ``Resolver.resolve_string()``.
Patch by Michael van Tellingen.
* GH#191: More accurate float serialisation in ``objectify.FloatElement``.
Patch by Holger Joukl.
* LP#1551797: Repair XSLT error logging. Patch by Marcus Brinkmann.
3.6.0 (2016-03-17)
==================
Features added
--------------
* GH#187: Now supports (only) version 5.x and later of PyPy.
Patch by Armin Rigo.
* GH#181: Direct support for ``.rnc`` files in `RelaxNG()` if ``rnc2rng``
is installed. Patch by Dirkjan Ochtman.
Bugs fixed
----------
* GH#189: Static builds honour FTP proxy configurations when downloading
the external libs. Patch by Youhei Sakurai.
* GH#186: Soupparser failed to process entities in Python 3.x.
Patch by Duncan Morris.
* GH#185: Rare encoding related ``TypeError`` on import was fixed.
Patch by Petr Demin.
3.5.0 (2015-11-13)
==================
Bugs fixed
----------
* Unicode string results failed XPath queries in PyPy.
* LP#1497051: HTML target parser failed to terminate on exceptions
and continued parsing instead.
* Deprecated API usage in doctestcompare.
3.5.0b1 (2015-09-18)
====================
Features added
--------------
* ``cleanup_namespaces()`` accepts a new argument ``keep_ns_prefixes``
that does not remove definitions of the provided prefix-namespace
mapping from the tree.
* ``cleanup_namespaces()`` accepts a new argument ``top_nsmap`` that
moves definitions of the provided prefix-namespace mapping to the
top of the tree.
* LP#1490451: ``Element`` objects gained a ``cssselect()`` method as
known from ``lxml.html``. Patch by Simon Sapin.
* API functions and methods behave and look more like Python functions,
which allows introspection on them etc. One side effect to be aware of
is that the functions now bind as methods when assigned to a class
variable. A quick fix is to wrap them in ``staticmethod()`` (as for
normal Python functions).
* ISO-Schematron support gained an option ``error_finder`` that allows
passing a filter function for picking validation errors from reports.
* LP#1243600: Elements in ``lxml.html`` gained a ``classes`` property
that provides a set-like interface to the ``class`` attribute.
Original patch by masklinn.
* LP#1341964: The soupparser now handles DOCTYPE declarations, comments
and processing instructions outside of the root element.
Patch by Olli Pottonen.
* LP#1421512: The ``docinfo`` of a tree was made editable to allow
setting and removing the public ID and system ID of the DOCTYPE.
Patch by Olli Pottonen.
* LP#1442427: More work-arounds for quirks and bugs in pypy and pypy3.
* ``lxml.html.soupparser`` now uses BeautifulSoup version 4 instead
of version 3 if available.
Bugs fixed
----------
* Memory errors that occur during tree adaptations (e.g. moving subtrees
to foreign documents) could leave the tree in a crash prone state.
* Calling ``process_children()`` in an XSLT extension element without
an ``output_parent`` argument failed with a ``TypeError``.
Fix by Jens Tröger.
* GH#162: Image data in HTML ``data`` URLs is considered safe and
no longer removed by ``lxml.html.clean`` JavaScript cleaner.
* GH#166: Static build could link libraries in wrong order.
* GH#172: Rely a bit more on libxml2 for encoding detection rather than
rolling our own in some cases. Patch by Olli Pottonen.
* GH#159: Validity checks for names and string content were tightened
to detect the use of illegal characters early. Patch by Olli Pottonen.
* LP#1421921: Comments/PIs before the DOCTYPE declaration were not
serialised. Patch by Olli Pottonen.
* LP#659367: Some HTML DOCTYPE declarations were not serialised.
Patch by Olli Pottonen.
* LP#1238503: lxml.doctestcompare is now consistent with stdlib's doctest
in how it uses ``+`` and ``-`` to refer to unexpected and missing output.
* Empty prefixes are explicitly rejected when a namespace mapping is used
with ElementPath to avoid hiding bugs in user code.
* Several problems with PyPy were fixed by switching to Cython 0.23.
3.4.4 (2015-04-25)
==================
Bugs fixed
----------
* An ElementTree compatibility test added in lxml 3.4.3 that failed in
Python 3.4+ was removed again.
3.4.3 (2015-04-15)
==================
Bugs fixed
----------
* Expression cache in ElementPath was ignored. Fix by Changaco.
* LP#1426868: Passing a default namespace and a prefixed namespace mapping
as nsmap into ``xmlfile.element()`` raised a ``TypeError``.
* LP#1421927: DOCTYPE system URLs were incorrectly quoted when containing
double quotes. Patch by Olli Pottonen.
* LP#1419354: meta-redirect URLs were incorrectly processed by
``iterlinks()`` if preceded by whitespace.
3.4.2 (2015-02-07)
==================
Bugs fixed
----------
* LP#1415907: Crash when creating an XMLSchema from a non-root element
of an XML document.
* LP#1369362: HTML cleaning failed when hitting processing instructions
with pseudo-attributes.
* ``CDATA()`` wrapped content was rejected for tail text.
* CDATA sections were not serialised as tail text of the top-level element.
3.4.1 (2014-11-20)
==================
Features added
--------------
* New ``htmlfile`` HTML generator to accompany the incremental ``xmlfile``
serialisation API. Patch by Burak Arslan.
Bugs fixed
----------
* ``lxml.sax.ElementTreeContentHandler`` did not initialise its superclass.
3.4.0 (2014-09-10)
==================
Features added
--------------
* ``xmlfile(buffered=False)`` disables output buffering and flushes the
content after each API operation (starting/ending element blocks or writes).
A new method ``xf.flush()`` can alternatively be used to explicitly flush
the output.
* ``lxml.html.document_fromstring`` has a new option ``ensure_head_body=True``
which will add an empty head and/or body element to the result document if
missing.
* ``lxml.html.iterlinks`` now returns links inside meta refresh tags.
* New ``XMLParser`` option ``collect_ids=False`` to disable ID hash table
creation. This can substantially speed up parsing of documents with many
different IDs that are not used.
* The parser uses per-document hash tables for XML IDs. This reduces the
load of the global parser dict and speeds up parsing for documents with
many different IDs.
* ``ElementTree.getelementpath(element)`` returns a structural ElementPath
expression for the given element, which can be used for lookups later.
* ``xmlfile()`` accepts a new argument ``close=True`` to close file(-like)
objects after writing to them. Before, ``xmlfile()`` only closed the file
if it had opened it internally.
* Allow "bytearray" type for ASCII text input.
Bugs fixed
----------
Other changes
-------------
* LP#400588: decoding errors have become hard errors even in recovery mode.
Previously, they could lead to an internal tree representation in a mixed
encoding state, which lead to very late errors or even silently incorrect
behaviour during tree traversal or serialisation.
* Requires Python 2.6, 2.7, 3.2 or later. No longer supports
Python 2.4, 2.5 and 3.1, use lxml 3.3.x for those.
* Requires libxml2 2.7.0 or later and libxslt 1.1.23 or later,
use lxml 3.3.x with older versions.
3.3.6 (2014-08-28)
==================
Bugs fixed
----------
* Prevent tree cycle creation when adding Elements as siblings.
* LP#1361948: crash when deallocating Element siblings without parent.
* LP#1354652: crash when traversing internally loaded documents in XSLT
extension functions.
3.3.5 (2014-04-18)
==================
Bugs fixed
----------
* HTML cleaning could fail to strip javascript links that mix control
characters into the link scheme.
3.3.4 (2014-04-03)
==================
Features added
--------------
* Source line numbers above 65535 are available on Elements when
using libxml2 2.9 or later.
Bugs fixed
----------
* ``lxml.html.fragment_fromstring()`` failed for bytes input in Py3.
Other changes
-------------
3.3.3 (2014-03-04)
==================
Bugs fixed
----------
* LP#1287118: Crash when using Element subtypes with ``__slots__``.
Other changes
-------------
* The internal classes ``_LogEntry`` and ``_Attrib`` can no longer be
subclassed from Python code.
3.3.2 (2014-02-26)
==================
Bugs fixed
----------
* The properties ``resolvers`` and ``version``, as well as the methods
``set_element_class_lookup()`` and ``makeelement()``, were lost from
``iterparse`` objects in 3.3.0.
* LP#1222132: instances of ``XMLSchema``, ``Schematron`` and ``RelaxNG``
did not clear their local ``error_log`` before running a validation.
* LP#1238500: lxml.doctestcompare mixed up "expected" and "actual" in
attribute values.
* Some file I/O tests were failing in MS-Windows due to non-portable temp
file usage. Initial patch by Gabi Davar.
* LP#910014: duplicate IDs in a document were not reported by DTD validation.
* LP#1185332: ``tostring(method="html")`` did not use HTML serialisation
semantics for trailing tail text. Initial patch by Sylvain Viollon.
* LP#1281139: ``.attrib`` value of Comments lost its mutation methods
in 3.3.0. Even though it is empty and immutable, it should still
provide the same interface as that returned for Elements.
3.3.1 (2014-02-12)
==================
Features added
--------------
Bugs fixed
----------
* LP#1014290: HTML documents parsed with ``parser.feed()`` failed to find
elements during tag iteration.
* LP#1273709: Building in PyPy failed due to missing support for
``PyUnicode_Compare()`` and ``PyByteArray_*()`` in PyPy's C-API.
* LP#1274413: Compilation in MSVC failed due to missing "stdint.h" standard
header file.
* LP#1274118: iterparse() failed to parse BOM prefixed files.
Other changes
-------------
3.3.0 (2014-01-26)
==================
Features added
--------------
Bugs fixed
----------
* The heuristic that distinguishes file paths from URLs was tightened
to produce less false negatives.
Other changes
-------------
3.3.0beta5 (2014-01-18)
=======================
Features added
--------------
* The PEP 393 unicode parsing support gained a fallback for wchar strings
which might still be somewhat common on Windows systems.
Bugs fixed
----------
* Several error handling problems were fixed throughout the code base that
could previously lead to exceptions being silently swallowed or not
properly reported.
* The C-API function ``appendChild()`` is now deprecated as it does not
propagate exceptions (its return type is ``void``). The new function
``appendChildToElement()`` was added as a safe replacement.
* Passing a string into ``fromstringlist()`` raises an exception instead of
parsing the string character by character.
Other changes
-------------
* Document cleanup code was simplified using the new GC features in
Cython 0.20.
3.3.0beta4 (2014-01-12)
=======================
Features added
--------------
Bugs fixed
----------
* The (empty) value returned by the ``attrib`` property of Entity and Comment
objects was mutable.
* Element class lookup wasn't available for the new pull parsers or when using
a custom parser target.
* Setting Element attributes on instantiation with both the ``attrib`` argument
and keyword arguments could modify the mapping passed as ``attrib``.
* LP#1266171: DTDs instantiated from internal/external subsets (i.e. through
the docinfo property) lost their attribute declarations.
Other changes
-------------
* Built with Cython 0.20pre (gitrev 012ae82eb) to prepare support for
Python 3.4.
3.3.0beta3 (2014-01-02)
=======================
Features added
--------------
* Unicode string parsing was optimised for Python 3.3 (PEP 393).
Bugs fixed
----------
* HTML parsing of Unicode strings could misdecode the input on some platforms.
* Crash in xmlfile() when closing open elements out of order in an error case.
Other changes
-------------
3.3.0beta2 (2013-12-20)
=======================
Features added
--------------
* ``iterparse()`` supports the ``recover`` option.
Bugs fixed
----------
* Crash in ``iterparse()`` for HTML parsing.
* Crash in target parsing with attributes.
Other changes
-------------
* The safety check in the read-only tree implementation (e.g. used by
``PythonElementClassLookup``) raises a more appropriate ``ReferenceError``
for illegal access after tree disposal instead of an ``AssertionError``.
This should only impact test code that specifically checks the original
behaviour.
3.3.0beta1 (2013-12-12)
=======================
Features added
--------------
* New option ``handle_failures`` in ``make_links_absolute()`` and
``resolve_base_href()`` (lxml.html) that enables ignoring or
discarding links that fail to parse as URLs.
* New parser classes ``XMLPullParser`` and ``HTMLPullParser`` for
incremental parsing, as implemented for ElementTree in Python 3.4.
* ``iterparse()`` enables recovery mode by default for HTML parsing
(``html=True``).
Bugs fixed
----------
* LP#1255132: crash when trying to run validation over non-Element (e.g.
comment or PI).
* Error messages in the log and in exception messages that originated
from libxml2 could accidentally be picked up from preceding warnings
instead of the actual error.
* The ``ElementMaker`` in lxml.objectify did not accept a dict as
argument for adding attributes to the element it's building. This
works as in lxml.builder now.
* LP#1228881: ``repr(XSLTAccessControl)`` failed in Python 3.
* Raise ``ValueError`` when trying to append an Element to itself or
to one of its own descendants, instead of running into an infinite
loop.
* LP#1206077: htmldiff discarded whitespace from the output.
* Compressed plain-text serialisation to file-like objects was broken.
* lxml.html.formfill: Fix textarea form filling.
The textarea used to be cleared before the new content was set,
which removed the name attribute.
Other changes
-------------
* Some basic API classes use freelists internally for faster
instantiation. This can speed up some ``iterparse()`` scenarios,
for example.
* ``iterparse()`` was rewritten to use the new ``*PullParser``
classes internally instead of being a parser itself.
3.2.5 (2014-01-02)
==================
Features added
--------------
Bugs fixed
----------
* Crash in xmlfile() when closing open elements out of order in an error case.
* Crash in target parsing with attributes.
* LP#1255132: crash when trying to run validation over non-Element (e.g.
comment or PI).
Other changes
-------------
3.2.4 (2013-11-07)
==================
Features added
--------------
Bugs fixed
----------
* Memory leak when creating an XPath evaluator in a thread.
* LP#1228881: ``repr(XSLTAccessControl)`` failed in Python 3.
* Raise ``ValueError`` when trying to append an Element to itself or
to one of its own descendants.
* LP#1206077: htmldiff discarded whitespace from the output.
* Compressed plain-text serialisation to file-like objects was broken.
Other changes
-------------
3.2.3 (2013-07-28)
==================
Bugs fixed
----------
* Fix support for Python 2.4 which was lost in 3.2.2.
3.2.2 (2013-07-28)
==================
Features added
--------------
Bugs fixed
----------
* LP#1185701: spurious XMLSyntaxError after finishing iterparse().
* Crash in lxml.objectify during xsi annotation.
Other changes
-------------
* Return values of user provided element class lookup methods are now
validated against the type of the XML node they represent to prevent
API class mismatches.
3.2.1 (2013-05-11)
==================
Features added
--------------
* The methods ``apply_templates()`` and ``process_children()`` of XSLT
extension elements have gained two new boolean options ``elements_only``
and ``remove_blank_text`` that discard either all strings or whitespace-only
strings from the result list.
Bugs fixed
----------
* When moving Elements to another tree, the namespace cleanup mechanism
no longer drops namespace prefixes from attributes for which it finds
a default namespace declaration, to prevent them from appearing as
unnamespaced attributes after serialisation.
* Returning non-type objects from a custom class lookup method could lead
to a crash.
* Instantiating and using subtypes of Comments and ProcessingInstructions
crashed.
Other changes
-------------
3.2.0 (2013-04-28)
==================
Features added
--------------
Bugs fixed
----------
* LP#690319: Leading whitespace could change the behaviour of the string
parsing functions in ``lxml.html``.
* LP#599318: The string parsing functions in ``lxml.html`` are more robust
in the face of uncommon HTML content like framesets or missing body tags.
Patch by Stefan Seelmann.
* LP#712941: I/O errors while trying to access files with paths that contain
non-ASCII characters could raise ``UnicodeDecodeError`` instead of properly
reporting the ``IOError``.
* LP#673205: Parsing from in-memory strings disabled network access in the
default parser and made subsequent attempts to parse from a URL fail.
* LP#971754: lxml.html.clean appends 'nofollow' to 'rel' attributes instead
of overwriting the current value.
* LP#715687: lxml.html.clean no longer discards scripts that are explicitly
allowed by the user provided whitelist. Patch by Christine Koppelt.
Other changes
-------------
3.1.2 (2013-04-12)
==================
Features added
--------------
Bugs fixed
----------
* LP#1136509: Passing attributes through the namespace-unaware API of
the sax bridge (i.e. the ``handler.startElement()`` method) failed
with a ``TypeError``. Patch by Mike Bayer.
* LP#1123074: Fix serialisation error in XSLT output when converting
the result tree to a Unicode string.
* GH#105: Replace illegal usage of ``xmlBufLength()`` in libxml2 2.9.0
by properly exported API function ``xmlBufUse()``.
Other changes
-------------
3.1.1 (2013-03-29)
==================
Features added
--------------
Bugs fixed
----------
* LP#1160386: Write access to ``lxml.html.FormElement.fields`` raised
an AttributeError in Py3.
* Illegal memory access during cleanup in incremental xmlfile writer.
Other changes
-------------
* The externally useless class ``lxml.etree._BaseParser`` was removed
from the module dict.
3.1.0 (2013-02-10)
==================
Features added
--------------
* GH#89: lxml.html.clean allows overriding the set of attributes that it
considers 'safe'. Patch by Francis Devereux.
Bugs fixed
----------
* LP#1104370: ``copy.copy(el.attrib)`` raised an exception. It now returns
a copy of the attributes as a plain Python dict.
* GH#95: When used with namespace prefixes, the ``el.find*()`` methods
always used the first namespace mapping that was provided for each
path expression instead of using the one that was actually passed
in for the current run.
* LP#1092521, GH#91: Fix undefined C symbol in Python runtimes compiled
without threading support. Patch by Ulrich Seidl.
Other changes
-------------
3.1beta1 (2012-12-21)
=====================
Features added
--------------
* New build-time option ``--with-unicode-strings`` for Python 2 that
makes the API always return Unicode strings for names and text
instead of byte strings for plain ASCII content.
* New incremental XML file writing API ``etree.xmlfile()``.
* E factory in lxml.objectify is callable to simplify the creation of
tags with non-identifier names without having to resort to getattr().
Bugs fixed
----------
* When starting from a non-namespaced element in lxml.objectify, searching
for a child without explicitly specifying a namespace incorrectly found
namespaced elements with the requested local name, instead of restricting
the search to non-namespaced children.
* GH#85: Deprecation warnings were fixed for Python 3.x.
* GH#33: lxml.html.fromstring() failed to accept bytes input in Py3.
* LP#1080792: Static build of libxml2 2.9.0 failed due to missing file.
Other changes
-------------
* The externally useless class ``_ObjectifyElementMakerCaller`` was
removed from the module API of lxml.objectify.
* LP#1075622: lxml.builder is faster for adding text to elements with
many children. Patch by Anders Hammarquist.
3.0.2 (2012-12-14)
==================
Features added
--------------
Bugs fixed
----------
* Fix crash during interpreter shutdown by switching to Cython 0.17.3 for building.
Other changes
-------------
3.0.1 (2012-10-14)
==================
Features added
--------------
Bugs fixed
----------
* LP#1065924: Element proxies could disappear during garbage collection
in PyPy without proper cleanup.
* GH#71: Failure to work with libxml2 2.6.x.
* LP#1065139: static MacOS-X build failed in Py3.
Other changes
-------------
3.0 (2012-10-08)
================
Features added
--------------
Bugs fixed
----------
* End-of-file handling was incorrect in iterparse() when reading from
a low-level C file stream and failed in libxml2 2.9.0 due to its
improved consistency checks.
Other changes
-------------
* The build no longer uses Cython by default unless the generated C files
are missing. To use Cython, pass the option "--with-cython". To ignore
the fatal build error when Cython is required but not available (e.g. to
run special setup.py commands that do not actually run a build), pass
"--without-cython".
3.0beta1 (2012-09-26)
=====================
Features added
--------------
* Python level access to (optional) libxml2 memory debugging features
to simplify debugging of memory leaks etc.
Bugs fixed
----------
* Fix a memory leak in XPath by switching to Cython 0.17.1.
* Some tests were adapted to work with PyPy.
Other changes
-------------
* The code was adapted to work with the upcoming libxml2 2.9.0 release.
3.0alpha2 (2012-08-23)
======================
Features added
--------------
* The ``.iter()`` method of elements now accepts ``tag`` arguments like
``"{*}name"`` to search for elements with a given local name in any
namespace. With this addition, all combinations of wildcards now work
as expected:
``"{ns}name"``, ``"{}name"``, ``"{*}name"``, ``"{ns}*"``, ``"{}*"``
and ``"{*}*"``. Note that ``"name"`` is equivalent to ``"{}name"``,
but ``"*"`` is ``"{*}*"``.
The same change applies to the ``.getiterator()``, ``.itersiblings()``,
``.iterancestors()``, ``.iterdescendants()``, ``.iterchildren()``
and ``.itertext()`` methods;the ``strip_attributes()``,
``strip_elements()`` and ``strip_tags()`` functions as well as the
``iterparse()`` class. Patch by Simon Sapin.
* C14N allows specifying the inclusive prefixes to be promoted
to top-level during exclusive serialisation.
Bugs fixed
----------
* Passing long Unicode strings into the ``feed()`` parser interface
failed to read the entire string.
Other changes
-------------
3.0alpha1 (2012-07-31)
======================
Features added
--------------
* Initial support for building in PyPy (through cpyext).
* DTD objects gained an API that allows read access to their
declarations.
* ``xpathgrep.py`` gained support for parsing line-by-line (e.g.
from grep output) and for surrounding the output with a new root
tag.
* ``E-factory`` in ``lxml.builder`` accepts subtypes of known data
types (such as string subtypes) when building elements around them.
* Tree iteration and ``iterparse()`` with a selective ``tag``
argument supports passing a set of tags. Tree nodes will be
returned by the iterators if they match any of the tags.
Bugs fixed
----------
* The ``.find*()`` methods in ``lxml.objectify`` no longer use XPath
internally, which makes them faster in many cases (especially when
short circuiting after a single or couple of elements) and fixes
some behavioural differences compared to ``lxml.etree``. Note that
this means that they no longer support arbitrary XPath expressions
but only the subset that the ``ElementPath`` language supports.
The previous implementation was also redundant with the normal
XPath support, which can be used as a replacement.
* ``el.find('*')`` could accidentally return a comment or processing
instruction that happened to be in the wrong spot. (Same for the
other ``.find*()`` methods.)
* The error logging is less intrusive and avoids a global setup where
possible.
* Fixed undefined names in html5lib parser.
* ``xpathgrep.py`` did not work in Python 3.
* ``Element.attrib.update()`` did not accept an ``attrib`` of
another Element as parameter.
* For subtypes of ``ElementBase`` that make the ``.text`` or ``.tail``
properties immutable (as in objectify, for example), inserting text
when creating Elements through the E-Factory feature of the class
constructor would fail with an exception, stating that the text
cannot be modified.
Other changes
--------------
* The code base was overhauled to properly use 'const' where the API
of libxml2 and libxslt requests it. This also has an impact on the
public C-API of lxml itself, as defined in ``etreepublic.pxd``, as
well as the provided declarations in the ``lxml/includes/`` directory.
Code that uses these declarations may have to be adapted. On the
plus side, this fixes several C compiler warnings, also for user
code, thus making it easier to spot real problems again.
* The functionality of "lxml.cssselect" was moved into a separate PyPI
package called "cssselect". To continue using it, you must install
that package separately. The "lxml.cssselect" module is still
available and provides the same interface, provided the "cssselect"
package can be imported at runtime.
* Element attributes passed in as an ``attrib`` dict or as keyword
arguments are now sorted by (namespaced) name before being created
to make their order predictable for serialisation and iteration.
Note that adding or deleting attributes afterwards does not take
that order into account, i.e. setting a new attribute appends it
after the existing ones.
* Several classes that are for internal use only were removed
from the ``lxml.etree`` module dict:
``_InputDocument, _ResolverRegistry, _ResolverContext, _BaseContext,
_ExsltRegExp, _IterparseContext, _TempStore, _ExceptionContext,
__ContentOnlyElement, _AttribIterator, _NamespaceRegistry,
_ClassNamespaceRegistry, _FunctionNamespaceRegistry,
_XPathFunctionNamespaceRegistry, _ParserDictionaryContext,
_FileReaderContext, _ParserContext, _PythonSaxParserTarget,
_TargetParserContext, _ReadOnlyProxy, _ReadOnlyPIProxy,
_ReadOnlyEntityProxy, _ReadOnlyElementProxy, _OpaqueNodeWrapper,
_OpaqueDocumentWrapper, _ModifyContentOnlyProxy,
_ModifyContentOnlyPIProxy, _ModifyContentOnlyEntityProxy,
_AppendOnlyElementProxy, _SaxParserContext, _FilelikeWriter,
_ParserSchemaValidationContext, _XPathContext,
_XSLTResolverContext, _XSLTContext, _XSLTQuotedStringParam``
* Several internal classes can no longer be inherited from:
``_InputDocument, _ResolverRegistry, _ExsltRegExp, _ElementUnicodeResult,
_IterparseContext, _TempStore, _AttribIterator, _ClassNamespaceRegistry,
_XPathFunctionNamespaceRegistry, _ParserDictionaryContext,
_FileReaderContext, _PythonSaxParserTarget, _TargetParserContext,
_ReadOnlyPIProxy, _ReadOnlyEntityProxy, _OpaqueDocumentWrapper,
_ModifyContentOnlyPIProxy, _ModifyContentOnlyEntityProxy,
_AppendOnlyElementProxy, _FilelikeWriter, _ParserSchemaValidationContext,
_XPathContext, _XSLTResolverContext, _XSLTContext, _XSLTQuotedStringParam,
_XSLTResultTree, _XSLTProcessingInstruction``
2.3.6 (2012-09-28)
==================
Features added
--------------
Bugs fixed
----------
* Passing long Unicode strings into the ``feed()`` parser interface
failed to read the entire string.
Other changes
--------------
2.3.5 (2012-07-31)
==================
Features added
--------------
Bugs fixed
----------
* Crash when merging text nodes in ``element.remove()``.
* Crash in sax/target parser when reporting empty doctype.
Other changes
--------------
2.3.4 (2012-03-26)
==================
Features added
--------------
Bugs fixed
----------
* Crash when building an nsmap (Element property) with empty
namespace URIs.
* Crash due to race condition when errors (or user messages) occur
during threaded XSLT processing.
* XSLT stylesheet compilation could ignore compilation errors.
Other changes
--------------
2.3.3 (2012-01-04)
==================
Features added
--------------
* ``lxml.html.tostring()`` gained new serialisation options
``with_tail`` and ``doctype``.
Bugs fixed
----------
* Fixed a crash when using ``iterparse()`` for HTML parsing and
requesting start events.
* Fixed parsing of more selectors in cssselect. Whitespace before
pseudo-elements and pseudo-classes is significant as it is a
descendant combinator.
"E :pseudo" should parse the same as "E \*:pseudo", not "E:pseudo".
Patch by Simon Sapin.
* lxml.html.diff no longer raises an exception when hitting
'img' tags without 'src' attribute.
Other changes
--------------
2.3.2 (2011-11-11)
==================
Features added
--------------
* ``lxml.objectify.deannotate()`` has a new boolean option
``cleanup_namespaces`` to remove the objectify namespace
declarations (and generally clean up the namespace declarations)
after removing the type annotations.
* ``lxml.objectify`` gained its own ``SubElement()`` function as a
copy of ``etree.SubElement`` to avoid an otherwise redundant import
of ``lxml.etree`` on the user side.
Bugs fixed
----------
* Fixed the "descendant" bug in cssselect a second time (after a first
fix in lxml 2.3.1). The previous change resulted in a serious
performance regression for the XPath based evaluation of the
translated expression. Note that this breaks the usage of some of
the generated XPath expressions as XSLT location paths that
previously worked in 2.3.1.
* Fixed parsing of some selectors in cssselect. Whitespace after combinators
">", "+" and "~" is now correctly ignored. Previously it was parsed as
a descendant combinator. For example, "div> .foo" was parsed the same as
"div>* .foo" instead of "div>.foo". Patch by Simon Sapin.
Other changes
--------------
2.3.1 (2011-09-25)
==================
Features added
--------------
* New option ``kill_tags`` in ``lxml.html.clean`` to remove specific
tags and their content (i.e. their whole subtree).
* ``pi.get()`` and ``pi.attrib`` on processing instructions to parse
pseudo-attributes from the text content of processing instructions.
* ``lxml.get_include()`` returns a list of include paths that can be
used to compile external C code against lxml.etree. This is
specifically required for statically linked lxml builds when code
needs to compile against the exact same header file versions as lxml
itself.
* ``Resolver.resolve_file()`` takes an additional option
``close_file`` that configures if the file(-like) object will be
closed after reading or not. By default, the file will be closed,
as the user is not expected to keep a reference to it.
Bugs fixed
----------
* HTML cleaning didn't remove 'data:' links.
* The html5lib parser integration now uses the 'official'
implementation in html5lib itself, which makes it work with newer
releases of the library.
* In ``lxml.sax``, ``endElementNS()`` could incorrectly reject a plain
tag name when the corresponding start event inferred the same plain
tag name to be in the default namespace.
* When an open file-like object is passed into ``parse()`` or
``iterparse()``, the parser will no longer close it after use. This
reverts a change in lxml 2.3 where all files would be closed. It is
the users responsibility to properly close the file(-like) object,
also in error cases.
* Assertion error in lxml.html.cleaner when discarding top-level elements.
* In lxml.cssselect, use the xpath 'A//B' (short for
'A/descendant-or-self::node()/B') instead of 'A/descendant::B' for
the css descendant selector ('A B'). This makes a few edge cases
like ``"div *:last-child"`` consistent with the selector behavior in
WebKit and Firefox, and makes more css expressions valid location
paths (for use in xsl:template match).
* In lxml.html, non-selected ```` tags no longer show up in the
collected form values.
* Adding/removing `` `` values to/from a multiple select form
field properly selects them and unselects them.
Other changes
--------------
* Static builds can specify the download directory with the
``--download-dir`` option.
2.3 (2011-02-06)
================
Features added
--------------
* When looking for children, ``lxml.objectify`` takes '{}tag' as
meaning an empty namespace, as opposed to the parent namespace.
Bugs fixed
----------
* When finished reading from a file-like object, the parser
immediately calls its ``.close()`` method.
* When finished parsing, ``iterparse()`` immediately closes the input
file.
* Work-around for libxml2 bug that can leave the HTML parser in a
non-functional state after parsing a severely broken document (fixed
in libxml2 2.7.8).
* ``marque`` tag in HTML cleanup code is correctly named ``marquee``.
Other changes
--------------
* Some public functions in the Cython-level C-API have more explicit
return types.
2.3beta1 (2010-09-06)
=====================
Features added
--------------
Bugs fixed
----------
* Crash in newer libxml2 versions when moving elements between
documents that had attributes on replaced XInclude nodes.
* ``XMLID()`` function was missing the optional ``parser`` and
``base_url`` parameters.
* Searching for wildcard tags in ``iterparse()`` was broken in Py3.
* ``lxml.html.open_in_browser()`` didn't work in Python 3 due to the
use of os.tempnam. It now takes an optional 'encoding' parameter.
Other changes
--------------
2.3alpha2 (2010-07-24)
======================
Features added
--------------
Bugs fixed
----------
* Crash in XSLT when generating text-only result documents with a
stylesheet created in a different thread.
Other changes
--------------
* ``repr()`` of Element objects shows the hex ID with leading 0x
(following ElementTree 1.3).
2.3alpha1 (2010-06-19)
======================
Features added
--------------
* Keyword argument ``namespaces`` in ``lxml.cssselect.CSSSelector()``
to pass a prefix-to-namespace mapping for the selector.
* New function ``lxml.etree.register_namespace(prefix, uri)`` that
globally registers a namespace prefix for a namespace that newly
created Elements in that namespace will use automatically. Follows
ElementTree 1.3.
* Support 'unicode' string name as encoding parameter in
``tostring()``, following ElementTree 1.3.
* Support 'c14n' serialisation method in ``ElementTree.write()`` and
``tostring()``, following ElementTree 1.3.
* The ElementPath expression syntax (``el.find*()``) was extended to
match the upcoming ElementTree 1.3 that will ship in the standard
library of Python 3.2/2.7. This includes extended support for
predicates as well as namespace prefixes (as known from XPath).
* During regular XPath evaluation, various ESXLT functions are
available within their namespace when using libxslt 1.1.26 or later.
* Support passing a readily configured logger instance into
``PyErrorLog``, instead of a logger name.
* On serialisation, the new ``doctype`` parameter can be used to
override the DOCTYPE (internal subset) of the document.
* New parameter ``output_parent`` to ``XSLTExtension.apply_templates()``
to append the resulting content directly to an output element.
* ``XSLTExtension.process_children()`` to process the content of the
XSLT extension element itself.
* ISO-Schematron support based on the de-facto Schematron reference
'skeleton implementation'.
* XSLT objects now take XPath object as ``__call__`` stylesheet
parameters.
* Enable path caching in ElementPath (``el.find*()``) to avoid parsing
overhead.
* Setting the value of a namespaced attribute always uses a prefixed
namespace instead of the default namespace even if both declare the
same namespace URI. This avoids serialisation problems when an
attribute from a default namespace is set on an element from a
different namespace.
* XSLT extension elements: support for XSLT context nodes other than
elements: document root, comments, processing instructions.
* Support for strings (in addition to Elements) in node-sets returned
by extension functions.
* Forms that lack an ``action`` attribute default to the base URL of
the document on submit.
* XPath attribute result strings have an ``attrname`` property.
* Namespace URIs get validated against RFC 3986 at the API level
(required by the XML namespace specification).
* Target parsers show their target object in the ``.target`` property
(compatible with ElementTree).
Bugs fixed
----------
* API is hardened against invalid proxy instances to prevent crashes
due to incorrectly instantiated Element instances.
* Prevent crash when instantiating ``CommentBase`` and friends.
* Export ElementTree compatible XML parser class as
``XMLTreeBuilder``, as it is called in ET 1.2.
* ObjectifiedDataElements in lxml.objectify were not hashable. They
now use the hash value of the underlying Python value (string,
number, etc.) to which they compare equal.
* Parsing broken fragments in lxml.html could fail if the fragment
contained an orphaned closing '' tag.
* Using XSLT extension elements around the root of the output document
crashed.
* ``lxml.cssselect`` did not distinguish between ``x[attr="val"]`` and
``x [attr="val"]`` (with a space). The latter now matches the
attribute independent of the element.
* Rewriting multiple links inside of HTML text content could end up
replacing unrelated content as replacements could impact the
reported position of subsequent matches. Modifications are now
simplified by letting the ``iterlinks()`` generator in ``lxml.html``
return links in reversed order if they appear inside the same text
node. Thus, replacements and link-internal modifications no longer
change the position of links reported afterwards.
* The ``.value`` attribute of ``textarea`` elements in lxml.html did
not represent the complete raw value (including child tags etc.). It
now serialises the complete content on read and replaces the
complete content by a string on write.
* Target parser didn't call ``.close()`` on the target object if
parsing failed. Now it is guaranteed that ``.close()`` will be
called after parsing, regardless of the outcome.
Other changes
-------------
* Official support for Python 3.1.2 and later.
* Static MS Windows builds can now download their dependencies
themselves.
* ``Element.attrib`` no longer uses a cyclic reference back to its
Element object. It therefore no longer requires the garbage
collector to clean up.
* Static builds include libiconv, in addition to libxml2 and libxslt.
2.2.8 (2010-09-02)
==================
Bugs fixed
----------
* Crash in newer libxml2 versions when moving elements between
documents that had attributes on replaced XInclude nodes.
* Import fix for urljoin in Python 3.1+.
2.2.7 (2010-07-24)
==================
Bugs fixed
----------
* Crash in XSLT when generating text-only result documents with a
stylesheet created in a different thread.
2.2.6 (2010-03-02)
==================
Bugs fixed
----------
* Fixed several Python 3 regressions by building with Cython 0.11.3.
2.2.5 (2010-02-28)
==================
Features added
--------------
* Support for running XSLT extension elements on the input root node
(e.g. in a template matching on "/").
Bugs fixed
----------
* Crash in XPath evaluation when reading smart strings from a document
other than the original context document.
* Support recent versions of html5lib by not requiring its
``XHTMLParser`` in ``htmlparser.py`` anymore.
* Manually instantiating the custom element classes in
``lxml.objectify`` could crash.
* Invalid XML text characters were not rejected by the API when they
appeared in unicode strings directly after non-ASCII characters.
* lxml.html.open_http_urllib() did not work in Python 3.
* The functions ``strip_tags()`` and ``strip_elements()`` in
``lxml.etree`` did not remove all occurrences of a tag in all cases.
* Crash in XSLT extension elements when the XSLT context node is not
an element.
2.2.4 (2009-11-11)
==================
Bugs fixed
----------
* Static build of libxml2/libxslt was broken.
2.2.3 (2009-10-30)
==================
Features added
--------------
Bugs fixed
----------
* The ``resolve_entities`` option did not work in the incremental feed
parser.
* Looking up and deleting attributes without a namespace could hit a
namespaced attribute of the same name instead.
* Late errors during calls to ``SubElement()`` (e.g. attribute related
ones) could leave a partially initialised element in the tree.
* Modifying trees that contain parsed entity references could result
in an infinite loop.
* ObjectifiedElement.__setattr__ created an empty-string child element when the
attribute value was rejected as a non-unicode/non-ascii string
* Syntax errors in ``lxml.cssselect`` could result in misleading error
messages.
* Invalid syntax in CSS expressions could lead to an infinite loop in
the parser of ``lxml.cssselect``.
* CSS special character escapes were not properly handled in
``lxml.cssselect``.
* CSS Unicode escapes were not properly decoded in ``lxml.cssselect``.
* Select options in HTML forms that had no explicit ``value``
attribute were not handled correctly. The HTML standard dictates
that their value is defined by their text content. This is now
supported by lxml.html.
* XPath raised a TypeError when finding CDATA sections. This is now
fully supported.
* Calling ``help(lxml.objectify)`` didn't work at the prompt.
* The ``ElementMaker`` in lxml.objectify no longer defines the default
namespaces when annotation is disabled.
* Feed parser failed to honour the 'recover' option on parse errors.
* Diverting the error logging to Python's logging system was broken.
Other changes
-------------
2.2.2 (2009-06-21)
==================
Features added
--------------
* New helper functions ``strip_attributes()``, ``strip_elements()``,
``strip_tags()`` in lxml.etree to remove attributes/subtrees/tags
from a subtree.
Bugs fixed
----------
* Namespace cleanup on subtree insertions could result in missing
namespace declarations (and potentially crashes) if the element
defining a namespace was deleted and the namespace was not used by
the top element of the inserted subtree but only in deeper subtrees.
* Raising an exception from a parser target callback didn't always
terminate the parser.
* Only {true, false, 1, 0} are accepted as the lexical representation for
BoolElement ({True, False, T, F, t, f} not any more), restoring lxml <= 2.0
behaviour.
Other changes
-------------
2.2.1 (2009-06-02)
==================
Features added
--------------
* Injecting default attributes into a document during XML Schema
validation (also at parse time).
* Pass ``huge_tree`` parser option to disable parser security
restrictions imposed by libxml2 2.7.
Bugs fixed
----------
* The script for statically building libxml2 and libxslt didn't work
in Py3.
* ``XMLSchema()`` also passes invalid schema documents on to libxml2
for parsing (which could lead to a crash before release 2.6.24).
Other changes
-------------
2.2 (2009-03-21)
================
Features added
--------------
* Support for ``standalone`` flag in XML declaration through
``tree.docinfo.standalone`` and by passing ``standalone=True/False``
on serialisation.
Bugs fixed
----------
* Crash when parsing an XML Schema with external imports from a
filename.
2.2beta4 (2009-02-27)
=====================
Features added
--------------
* Support strings and instantiable Element classes as child arguments
to the constructor of custom Element classes.
* GZip compression support for serialisation to files and file-like
objects.
Bugs fixed
----------
* Deep-copying an ElementTree copied neither its sibling PIs and
comments nor its internal/external DTD subsets.
* Soupparser failed on broken attributes without values.
* Crash in XSLT when overwriting an already defined attribute using
``xsl:attribute``.
* Crash bug in exception handling code under Python 3. This was due
to a problem in Cython, not lxml itself.
* ``lxml.html.FormElement._name()`` failed for non top-level forms.
* ``TAG`` special attribute in constructor of custom Element classes
was evaluated incorrectly.
Other changes
-------------
* Official support for Python 3.0.1.
* ``Element.findtext()`` now returns an empty string instead of None
for Elements without text content.
2.2beta3 (2009-02-17)
=====================
Features added
--------------
* ``XSLT.strparam()`` class method to wrap quoted string parameters
that require escaping.
Bugs fixed
----------
* Memory leak in XPath evaluators.
* Crash when parsing indented XML in one thread and merging it with
other documents parsed in another thread.
* Setting the ``base`` attribute in ``lxml.objectify`` from a unicode
string failed.
* Fixes following changes in Python 3.0.1.
* Minor fixes for Python 3.
Other changes
-------------
* The global error log (which is copied into the exception log) is now
local to a thread, which fixes some race conditions.
* More robust error handling on serialisation.
2.2beta2 (2009-01-25)
=====================
Bugs fixed
----------
* Potential memory leak on exception handling. This was due to a
problem in Cython, not lxml itself.
* ``iter_links`` (and related link-rewriting functions) in
``lxml.html`` would interpret CSS like ``url("link")`` incorrectly
(treating the quotation marks as part of the link).
* Failing import on systems that have an ``io`` module.
2.1.5 (2009-01-06)
==================
Bugs fixed
----------
* Potential memory leak on exception handling. This was due to a
problem in Cython, not lxml itself.
* Failing import on systems that have an ``io`` module.
2.2beta1 (2008-12-12)
=====================
Features added
--------------
* Allow ``lxml.html.diff.htmldiff`` to accept Element objects, not
just HTML strings.
Bugs fixed
----------
* Crash when using an XPath evaluator in multiple threads.
* Fixed missing whitespace before ``Link:...`` in ``lxml.html.diff``.
Other changes
-------------
* Export ``lxml.html.parse``.
2.1.4 (2008-12-12)
==================
Bugs fixed
----------
* Crash when using an XPath evaluator in multiple threads.
2.0.11 (2008-12-12)
===================
Bugs fixed
----------
* Crash when using an XPath evaluator in multiple threads.
2.2alpha1 (2008-11-23)
======================
Features added
--------------
* Support for XSLT result tree fragments in XPath/XSLT extension
functions.
* QName objects have new properties ``namespace`` and ``localname``.
* New options for exclusive C14N and C14N without comments.
* Instantiating a custom Element classes creates a new Element.
Bugs fixed
----------
* XSLT didn't inherit the parse options of the input document.
* 0-bytes could slip through the API when used inside of Unicode
strings.
* With ``lxml.html.clean.autolink``, links with balanced parenthesis,
that end in a parenthesis, will be linked in their entirety (typical
with Wikipedia links).
Other changes
-------------
2.1.3 (2008-11-17)
==================
Features added
--------------
Bugs fixed
----------
* Ref-count leaks when lxml enters a try-except statement while an
outside exception lives in sys.exc_*(). This was due to a problem in
Cython, not lxml itself.
* Parser Unicode decoding errors could get swallowed by other
exceptions.
* Name/import errors in some Python modules.
* Internal DTD subsets that did not specify a system or public ID were
not serialised and did not appear in the docinfo property of
ElementTrees.
* Fix a pre-Py3k warning when parsing from a gzip file in Py2.6.
* Test suite fixes for libxml2 2.7.
* Resolver.resolve_string() did not work for non-ASCII byte strings.
* Resolver.resolve_file() was broken.
* Overriding the parser encoding didn't work for many encodings.
Other changes
-------------
2.0.10 (2008-11-17)
===================
Bugs fixed
----------
* Ref-count leaks when lxml enters a try-except statement while an
outside exception lives in sys.exc_*(). This was due to a problem in
Cython, not lxml itself.
2.1.2 (2008-09-05)
==================
Features added
--------------
* lxml.etree now tries to find the absolute path name of files when
parsing from a file-like object. This helps custom resolvers when
resolving relative URLs, as lixbml2 can prepend them with the path
of the source document.
Bugs fixed
----------
* Memory problem when passing documents between threads.
* Target parser did not honour the ``recover`` option and raised an
exception instead of calling ``.close()`` on the target.
Other changes
-------------
2.0.9 (2008-09-05)
==================
Bugs fixed
----------
* Memory problem when passing documents between threads.
* Target parser did not honour the ``recover`` option and raised an
exception instead of calling ``.close()`` on the target.
2.1.1 (2008-07-24)
==================
Features added
--------------
Bugs fixed
----------
* Crash when parsing XSLT stylesheets in a thread and using them in
another.
* Encoding problem when including text with ElementInclude under
Python 3.
Other changes
-------------
2.0.8 (2008-07-24)
==================
Features added
--------------
* ``lxml.html.rewrite_links()`` strips links to work around documents
with whitespace in URL attributes.
Bugs fixed
----------
* Crash when parsing XSLT stylesheets in a thread and using them in
another.
* CSS selector parser dropped remaining expression after a function
with parameters.
Other changes
-------------
2.1 (2008-07-09)
================
Features added
--------------
* Smart strings can be switched off in XPath (``smart_strings``
keyword option).
* ``lxml.html.rewrite_links()`` strips links to work around documents
with whitespace in URL attributes.
Bugs fixed
----------
* Custom resolvers were not used for XMLSchema includes/imports and
XInclude processing.
* CSS selector parser dropped remaining expression after a function
with parameters.
Other changes
-------------
* ``objectify.enableRecursiveStr()`` was removed, use
``objectify.enable_recursive_str()`` instead
* Speed-up when running XSLTs on documents from other threads
2.0.7 (2008-06-20)
==================
Features added
--------------
* Pickling ``ElementTree`` objects in lxml.objectify.
Bugs fixed
----------
* Descending dot-separated classes in CSS selectors were not resolved
correctly.
* ``ElementTree.parse()`` didn't handle target parser result.
* Potential threading problem in XInclude.
* Crash in Element class lookup classes when the __init__() method of
the super class is not called from Python subclasses.
Other changes
-------------
* Non-ASCII characters in attribute values are no longer escaped on
serialisation.
2.1beta3 (2008-06-19)
=====================
Features added
--------------
* Major overhaul of ``tools/xpathgrep.py`` script.
* Pickling ``ElementTree`` objects in lxml.objectify.
* Support for parsing from file-like objects that return unicode
strings.
* New function ``etree.cleanup_namespaces(el)`` that removes unused
namespace declarations from a (sub)tree (experimental).
* XSLT results support the buffer protocol in Python 3.
* Polymorphic functions in ``lxml.html`` that accept either a tree or
a parsable string will return either a UTF-8 encoded byte string, a
unicode string or a tree, based on the type of the input.
Previously, the result was always a byte string or a tree.
* Support for Python 2.6 and 3.0 beta.
* File name handling now uses a heuristic to convert between byte
strings (usually filenames) and unicode strings (usually URLs).
* Parsing from a plain file object frees the GIL under Python 2.x.
* Running ``iterparse()`` on a plain file (or filename) frees the GIL
on reading under Python 2.x.
* Conversion functions ``html_to_xhtml()`` and ``xhtml_to_html()`` in
lxml.html (experimental).
* Most features in lxml.html work for XHTML namespaced tag names
(experimental).
Bugs fixed
----------
* ``ElementTree.parse()`` didn't handle target parser result.
* Crash in Element class lookup classes when the __init__() method of
the super class is not called from Python subclasses.
* A number of problems related to unicode/byte string conversion of
filenames and error messages were fixed.
* Building on MacOS-X now passes the "flat_namespace" option to the C
compiler, which reportedly prevents build quirks and crashes on this
platform.
* Windows build was broken.
* Rare crash when serialising to a file object with certain encodings.
Other changes
-------------
* Non-ASCII characters in attribute values are no longer escaped on
serialisation.
* Passing non-ASCII byte strings or invalid unicode strings as .tag,
namespaces, etc. will result in a ValueError instead of an
AssertionError (just like the tag well-formedness check).
* Up to several times faster attribute access (i.e. tree traversal) in
lxml.objectify.
2.0.6 (2008-05-31)
==================
Features added
--------------
Bugs fixed
----------
* Incorrect evaluation of ``el.find("tag[child]")``.
* Windows build was broken.
* Moving a subtree from a document created in one thread into a
document of another thread could crash when the rest of the source
document is deleted while the subtree is still in use.
* Rare crash when serialising to a file object with certain encodings.
Other changes
-------------
* lxml should now build without problems on MacOS-X.
2.1beta2 (2008-05-02)
=====================
Features added
--------------
* All parse functions in lxml.html take a ``parser`` keyword argument.
* lxml.html has a new parser class ``XHTMLParser`` and a module
attribute ``xhtml_parser`` that provide XML parsers that are
pre-configured for the lxml.html package.
Bugs fixed
----------
* Moving a subtree from a document created in one thread into a
document of another thread could crash when the rest of the source
document is deleted while the subtree is still in use.
* Passing an nsmap when creating an Element will no longer strip
redundantly defined namespace URIs. This prevented the definition
of more than one prefix for a namespace on the same Element.
Other changes
-------------
* If the default namespace is redundantly defined with a prefix on the
same Element, the prefix will now be preferred for subelements and
attributes. This allows users to work around a problem in libxml2
where attributes from the default namespace could serialise without
a prefix even when they appear on an Element with a different
namespace (i.e. they would end up in the wrong namespace).
2.0.5 (2008-05-01)
==================
Features added
--------------
Bugs fixed
----------
* Resolving to a filename in custom resolvers didn't work.
* lxml did not honour libxslt's second error state "STOPPED", which
let some XSLT errors pass silently.
* Memory leak in Schematron with libxml2 >= 2.6.31.
Other changes
-------------
2.1beta1 (2008-04-15)
=====================
Features added
--------------
* Error logging in Schematron (requires libxml2 2.6.32 or later).
* Parser option ``strip_cdata`` for normalising or keeping CDATA
sections. Defaults to ``True`` as before, thus replacing CDATA
sections by their text content.
* ``CDATA()`` factory to wrap string content as CDATA section.
Bugs fixed
----------
* Resolving to a filename in custom resolvers didn't work.
* lxml did not honour libxslt's second error state "STOPPED", which
let some XSLT errors pass silently.
* Memory leak in Schematron with libxml2 >= 2.6.31.
* lxml.etree accepted non well-formed namespace prefix names.
Other changes
-------------
* Major cleanup in internal ``moveNodeToDocument()`` function, which
takes care of namespace cleanup when moving elements between
different namespace contexts.
* New Elements created through the ``makeelement()`` method of an HTML
parser or through lxml.html now end up in a new HTML document
(doctype HTML 4.01 Transitional) instead of a generic XML document.
This mostly impacts the serialisation and the availability of a DTD
context.
2.0.4 (2008-04-13)
==================
Features added
--------------
Bugs fixed
----------
* Hanging thread in conjunction with GTK threading.
* Crash bug in iterparse when moving elements into other documents.
* HTML elements' ``.cssselect()`` method was broken.
* ``ElementTree.find*()`` didn't accept QName objects.
Other changes
-------------
2.1alpha1 (2008-03-27)
======================
Features added
--------------
* New event types 'comment' and 'pi' in ``iterparse()``.
* ``XSLTAccessControl`` instances have a property ``options`` that
returns a dict of access configuration options.
* Constant instances ``DENY_ALL`` and ``DENY_WRITE`` on
``XSLTAccessControl`` class.
* Extension elements for XSLT (experimental!)
* ``Element.base`` property returns the xml:base or HTML base URL of
an Element.
* ``docinfo.URL`` property is writable.
Bugs fixed
----------
* Default encoding for plain text serialisation was different from
that of XML serialisation (UTF-8 instead of ASCII).
Other changes
-------------
* Minor API speed-ups.
* The benchmark suite now uses tail text in the trees, which makes the
absolute numbers incomparable to previous results.
* Generating the HTML documentation now requires Pygments_, which is
used to enable syntax highlighting for the doctest examples.
.. _Pygments: http://pygments.org/
Most long-time deprecated functions and methods were removed:
- ``etree.clearErrorLog()``, use ``etree.clear_error_log()``
- ``etree.useGlobalPythonLog()``, use
``etree.use_global_python_log()``
- ``etree.ElementClassLookup.setFallback()``, use
``etree.ElementClassLookup.set_fallback()``
- ``etree.getDefaultParser()``, use ``etree.get_default_parser()``
- ``etree.setDefaultParser()``, use ``etree.set_default_parser()``
- ``etree.setElementClassLookup()``, use
``etree.set_element_class_lookup()``
Note that ``parser.setElementClassLookup()`` has not been removed
yet, although ``parser.set_element_class_lookup()`` should be used
instead.
- ``xpath_evaluator.registerNamespace()``, use
``xpath_evaluator.register_namespace()``
- ``xpath_evaluator.registerNamespaces()``, use
``xpath_evaluator.register_namespaces()``
- ``objectify.setPytypeAttributeTag``, use
``objectify.set_pytype_attribute_tag``
- ``objectify.setDefaultParser()``, use
``objectify.set_default_parser()``
2.0.3 (2008-03-26)
==================
Features added
--------------
* soupparser.parse() allows passing keyword arguments on to
BeautifulSoup.
* ``fromstring()`` method in ``lxml.html.soupparser``.
Bugs fixed
----------
* ``lxml.html.diff`` didn't treat empty tags properly (e.g.,
````
tags weren't recognized. (Note: plugin-specific link parameters
still aren't recognized.) Also, the ```` tag, though not
standard, is now included in ``lxml.html.defs.special_inline_tags``.
* Using custom resolvers on XSLT stylesheets parsed from a string
could request ill-formed URLs.
* With ``lxml.doctestcompare`` if you do ```` in your
output, it will then be namespace-neutral (before the ellipsis was
treated as a real namespace).
* AttributeError in feed parser on parse errors
* XML feed parser setup problem
* Type annotation for unicode strings in ``DataElement()``
* lxml failed to serialise namespace declarations of elements other than the
root node of a tree
* Race condition in XSLT where the resolver context leaked between concurrent
XSLT calls
* lxml.etree did not check tag/attribute names
* The XML parser did not report undefined entities as error
* The text in exceptions raised by XML parsers, validators and XPath
evaluators now reports the first error that occurred instead of the last
* Passing '' as XPath namespace prefix did not raise an error
* Thread safety in XPath evaluators
Other changes
-------------
* Exceptions carry only the part of the error log that is related to
the operation that caused the error.
* ``XMLSchema()`` and ``RelaxNG()`` now enforce passing the source
file/filename through the ``file`` keyword argument.
* The test suite now skips most doctests under Python 2.3.
* ``make clean`` no longer removes the .c files (use ``make
realclean`` instead)
* Minor performance tweaks for Element instantiation and subelement
creation
* Various places in the XPath, XSLT and iteration APIs now require
keyword-only arguments.
* The argument order in ``element.itersiblings()`` was changed to
match the order used in all other iteration methods. The second
argument ('preceding') is now a keyword-only argument.
* The ``getiterator()`` method on Elements and ElementTrees was
reverted to return an iterator as it did in lxml 1.x. The ET API
specification allows it to return either a sequence or an iterator,
and it traditionally returned a sequence in ET and an iterator in
lxml. However, it is now deprecated in favour of the ``iter()``
method, which should be used in new code wherever possible.
* The 'pretty printed' serialisation of ElementTree objects now
inserts newlines at the root level between processing instructions,
comments and the root tag.
* A 'pretty printed' serialisation is now terminated with a newline.
* Second argument to ``lxml.etree.Extension()`` helper is no longer
required, third argument is now a keyword-only argument ``ns``.
* ``lxml.html.tostring`` takes an ``encoding`` argument.
* The module source files were renamed to "lxml.*.pyx", such as
"lxml.etree.pyx". This was changed for consistency with the way
Pyrex commonly handles package imports. The main effect is that
classes now know about their fully qualified class name, including
the package name of their module.
* Keyword-only arguments in some API functions, especially in the
parsers and serialisers.
* Tag name validation in lxml.etree (and lxml.html) now distinguishes
between HTML tags and XML tags based on the parser that was used to
parse or create them. HTML tags no longer reject any non-ASCII
characters in tag names but only spaces and the special characters
``<>&/"'``.
* lxml.etree now emits a warning if you use XPath with libxml2 2.6.27
(which can crash on certain XPath errors)
* Type annotation in objectify now preserves the already annotated type by
default to prevent losing type information that is already there.
* ``element.getiterator()`` returns a list, use ``element.iter()`` to retrieve
an iterator (ElementTree 1.3 compatible behaviour)
* objectify.PyType for None is now called "NoneType"
* ``el.getiterator()`` renamed to ``el.iter()``, following ElementTree 1.3 -
original name is still available as alias
* In the public C-API, ``findOrBuildNodeNs()`` was replaced by the more
generic ``findOrBuildNodeNsPrefix``
* Major refactoring in XPath/XSLT extension function code
* Network access in parsers disabled by default
1.3.6 (2007-10-29)
==================
Bugs fixed
----------
* Backported decref crash fix from 2.0
* Well hidden free-while-in-use crash bug in ObjectPath
Other changes
-------------
* The test suites now run ``gc.collect()`` in the ``tearDown()``
methods. While this makes them take a lot longer to run, it also
makes it easier to link a specific test to garbage collection
problems that would otherwise appear in later tests.
1.3.5 (2007-10-22)
==================
Features added
--------------
Bugs fixed
----------
* lxml.etree could crash when adding more than 10000 namespaces to a
document
* lxml failed to serialise namespace declarations of elements other
than the root node of a tree
1.3.4 (2007-08-30)
==================
Features added
--------------
* The ``ElementMaker`` in ``lxml.builder`` now accepts the keyword arguments
``namespace`` and ``nsmap`` to set a namespace and nsmap for the Elements it
creates.
* The ``docinfo`` on ElementTree objects has new properties ``internalDTD``
and ``externalDTD`` that return a DTD object for the internal or external
subset of the document respectively.
* Serialising an ElementTree now includes any internal DTD subsets that are
part of the document, as well as comments and PIs that are siblings of the
root node.
Bugs fixed
----------
* Parsing with the ``no_network`` option could fail
Other changes
-------------
* lxml now raises a TagNameWarning about tag names containing ':' instead of
an Error as 1.3.3 did. The reason is that a number of projects currently
misuse the previous lack of tag name validation to generate namespace
prefixes without declaring namespaces. Apart from the danger of generating
broken XML this way, it also breaks most of the namespace-aware tools in
XML, including XPath, XSLT and validation. lxml 1.3.x will continue to
support this bug with a Warning, while lxml 2.0 will be strict about
well-formed tag names (not only regarding ':').
* Serialising an Element no longer includes its comment and PI siblings (only
ElementTree serialisation includes them).
1.3.3 (2007-07-26)
==================
Features added
--------------
* ElementTree compatible parser ``ETCompatXMLParser`` strips processing
instructions and comments while parsing XML
* Parsers now support stripping PIs (keyword argument 'remove_pis')
* ``etree.fromstring()`` now supports parsing both HTML and XML, depending on
the parser you pass.
* Support ``base_url`` keyword argument in ``HTML()`` and ``XML()``
Bugs fixed
----------
* Parsing from Python Unicode strings failed on some platforms
* ``Element()`` did not raise an exception on tag names containing ':'
* ``Element.getiterator(tag)`` did not accept ``Comment`` and
``ProcessingInstruction`` as tags. It also accepts ``Element`` now.
1.3.2 (2007-07-03)
==================
Features added
--------------
Bugs fixed
----------
* "deallocating None" crash bug
1.3.1 (2007-07-02)
==================
Features added
--------------
* objectify.DataElement now supports setting values from existing data
elements (not just plain Python types) and reuses defined namespaces etc.
* E-factory support for lxml.objectify (``objectify.E``)
Bugs fixed
----------
* Better way to prevent crashes in Element proxy cleanup code
* objectify.DataElement didn't set up None value correctly
* objectify.DataElement didn't check the value against the provided type hints
* Reference-counting bug in ``Element.attrib.pop()``
1.3 (2007-06-24)
================
Features added
--------------
* Module ``lxml.pyclasslookup`` module implements an Element class lookup
scheme that can access the entire tree in read-only mode to help determining
a suitable Element class
* Parsers take a ``remove_comments`` keyword argument that skips over comments
* ``parse()`` function in ``objectify``, corresponding to ``XML()`` etc.
* ``Element.addnext(el)`` and ``Element.addprevious(el)`` methods to support
adding processing instructions and comments around the root node
* ``Element.attrib`` was missing ``clear()`` and ``pop()`` methods
* Extended type annotation in objectify: cleaner annotation namespace setup
plus new ``deannotate()`` function
* Support for custom Element class instantiation in lxml.sax: passing a
``makeelement`` function to the ElementTreeContentHandler will reuse the
lookup context of that function
* '.' represents empty ObjectPath (identity)
* ``Element.values()`` to accompany the existing ``.keys()`` and ``.items()``
* ``collectAttributes()`` C-function to build a list of attribute
keys/values/items for a libxml2 node
* ``DTD`` validator class (like ``RelaxNG`` and ``XMLSchema``)
* HTML generator helpers by Fredrik Lundh in ``lxml.htmlbuilder``
* ``ElementMaker`` XML generator by Fredrik Lundh in ``lxml.builder.E``
* Support for pickling ``objectify.ObjectifiedElement`` objects to XML
* ``update()`` method on Element.attrib
* Optimised replacement for libxml2's _xmlReconsiliateNs(). This allows lxml
a better handling of namespaces when moving elements between documents.
Bugs fixed
----------
* Removing Elements from a tree could make them lose their namespace
declarations
* ``ElementInclude`` didn't honour base URL of original document
* Replacing the children slice of an Element would cut off the tails of the
original children
* ``Element.getiterator(tag)`` did not accept ``Comment`` and
``ProcessingInstruction`` as tags
* API functions now check incoming strings for XML conformity. Zero bytes or
low ASCII characters are no longer accepted (AssertionError).
* XSLT parsing failed to pass resolver context on to imported documents
* passing '' as namespace prefix in nsmap could be passed through to libxml2
* Objectify couldn't handle prefixed XSD type names in ``xsi:type``
* More ET compatible behaviour when writing out XML declarations or not
* More robust error handling in ``iterparse()``
* Documents lost their top-level PIs and comments on serialisation
* lxml.sax failed on comments and PIs. Comments are now properly ignored and
PIs are copied.
* Possible memory leaks in namespace handling when moving elements between
documents
Other changes
-------------
* major restructuring in the documentation
1.2.1 (2007-02-27)
==================
Bugs fixed
----------
* Build fixes for MS compiler
* Item assignments to special names like ``element["text"]`` failed
* Renamed ObjectifiedDataElement.__setText() to _setText() to make it easier
to access
* The pattern for attribute names in ObjectPath was too restrictive
1.2 (2007-02-20)
================
Features added
--------------
* Rich comparison of QName objects
* Support for regular expressions in benchmark selection
* get/set emulation (not .attrib!) for attributes on processing instructions
* ElementInclude Python module for ElementTree compatible XInclude processing
that honours custom resolvers registered with the source document
* ElementTree.parser property holds the parser used to parse the document
* setup.py has been refactored for greater readability and flexibility
* --rpath flag to setup.py to induce automatic linking-in of dynamic library
runtime search paths has been renamed to --auto-rpath. This makes it
possible to pass an --rpath directly to distutils; previously this was being
shadowed.
Bugs fixed
----------
* Element instantiation now uses locks to prevent race conditions with threads
* ElementTree.write() did not raise an exception when the file was not writable
* Error handling could crash under Python <= 2.4.1 - fixed by disabling thread
support in these environments
* Element.find*() did not accept QName objects as path
Other changes
-------------
* code cleanup: redundant _NodeBase super class merged into _Element class
Note: although the impact should be zero in most cases, this change breaks
the compatibility of the public C-API
1.1.2 (2006-10-30)
==================
Features added
--------------
* Data elements in objectify support repr(), which is now used by dump()
* Source distribution now ships with a patched Pyrex
* New C-API function makeElement() to create new elements with text,
tail, attributes and namespaces
* Reuse original parser flags for XInclude
* Simplified support for handling XSLT processing instructions
Bugs fixed
----------
* Parser resources were not freed before the next parser run
* Open files and XML strings returned by Python resolvers were not
closed/freed
* Crash in the IDDict returned by XMLDTDID
* Copying Comments and ProcessingInstructions failed
* Memory leak for external URLs in _XSLTProcessingInstruction.parseXSL()
* Memory leak when garbage collecting tailed root elements
* HTML script/style content was not propagated to .text
* Show text xincluded between text nodes correctly in .text and .tail
* 'integer * objectify.StringElement' operation was not supported
1.1.1 (2006-09-21)
==================
Features added
--------------
* XSLT profiling support (``profile_run`` keyword)
* countchildren() method on objectify.ObjectifiedElement
* Support custom elements for tree nodes in lxml.objectify
Bugs fixed
----------
* lxml.objectify failed to support long data values (e.g., "123L")
* Error messages from XSLT did not reach ``XSLT.error_log``
* Factories objectify.Element() and objectify.DataElement() were missing
``attrib`` and ``nsmap`` keyword arguments
* Changing the default parser in lxml.objectify did not update the factories
Element() and DataElement()
* Let lxml.objectify.Element() always generate tree elements (not data
elements)
* Build under Windows failed ('\0' bug in patched Pyrex version)
1.1 (2006-09-13)
================
Features added
--------------
* Comments and processing instructions return '' and
'' for repr()
* Parsers are now the preferred (and default) place where element class lookup
schemes should be registered. Namespace lookup is no longer supported by
default.
* Support for Python 2.5 beta
* Unlock the GIL for deep copying documents and for XPath()
* New ``compact`` keyword argument for parsing read-only documents
* Support for parser options in iterparse()
* The ``namespace`` axis is supported in XPath and returns (prefix, URI)
tuples
* The XPath expression "/" now returns an empty list instead of raising an
exception
* XML-Object API on top of lxml (lxml.objectify)
* Customizable Element class lookup:
* different pre-implemented lookup mechanisms
* support for externally provided lookup functions
* Support for processing instructions (ET-like, not compatible)
* Public C-level API for independent extension modules
* Module level ``iterwalk()`` function as 'iterparse' for trees
* Module level ``iterparse()`` function similar to ElementTree (see
documentation for differences)
* Element.nsmap property returns a mapping of all namespace prefixes known at
the Element to their namespace URI
* Reentrant threading support in RelaxNG, XMLSchema and XSLT
* Threading support in parsers and serializers:
* All in-memory operations (tostring, parse(StringIO), etc.) free the GIL
* File operations (on file names) free the GIL
* Reading from file-like objects frees the GIL and reacquires it for reading
* Serialisation to file-like objects is single-threaded (high lock overhead)
* Element iteration over XPath axes:
* Element.iterdescendants() iterates over the descendants of an element
* Element.iterancestors() iterates over the ancestors of an element (from
parent to parent)
* Element.itersiblings() iterates over either the following or preceding
siblings of an element
* Element.iterchildren() iterates over the children of an element in either
direction
* All iterators support the ``tag`` keyword argument to restrict the
generated elements
* Element.getnext() and Element.getprevious() return the direct siblings of an
element
Bugs fixed
----------
* filenames with local 8-bit encoding were not supported
* 1.1beta did not compile under Python 2.3
* ignore unknown 'pyval' attribute values in objectify
* objectify.ObjectifiedElement.addattr() failed to accept Elements and Lists
* objectify.ObjectPath.setattr() failed to accept Elements and Lists
* XPathSyntaxError now inherits from XPathError
* Threading race conditions in RelaxNG and XMLSchema
* Crash when mixing elements from XSLT results into other trees, concurrent
XSLT is only allowed when the stylesheet was parsed in the main thread
* The EXSLT ``regexp:match`` function now works as defined (except for some
differences in the regular expression syntax)
* Setting element.text to '' returned None on request, not the empty string
* ``iterparse()`` could crash on long XML files
* Creating documents no longer copies the parser for later URL resolving. For
performance reasons, only a reference is kept. Resolver updates on the
parser will now be reflected by documents that were parsed before the
change. Although this should rarely become visible, it is a behavioral
change from 1.0.
1.0.4 (2006-09-09)
==================
Features added
--------------
* List-like ``Element.extend()`` method
Bugs fixed
----------
* Crash in tail handling in ``Element.replace()``
1.0.3 (2006-08-08)
==================
Features added
--------------
* Element.replace(old, new) method to replace a subelement by another one
Bugs fixed
----------
* Crash when mixing elements from XSLT results into other trees
* Copying/deepcopying did not work for ElementTree objects
* Setting an attribute to a non-string value did not raise an exception
* Element.remove() deleted the tail text from the removed Element
1.0.2 (2006-06-27)
==================
Features added
--------------
* Support for setting a custom default Element class as opposed to namespace
specific classes (which still override the default class)
Bugs fixed
----------
* Rare exceptions in Python list functions were not handled
* Parsing accepted unicode strings with XML encoding declaration in certain
cases
* Parsing 8-bit encoded strings from StringIO objects raised an exception
* Module function ``initThread()`` was removed - useless (and never worked)
* XSLT and parser exception messages include the error line number
1.0.1 (2006-06-09)
==================
Features added
--------------
* Repeated calls to Element.attrib now efficiently return the same instance
Bugs fixed
----------
* Document deallocation could crash in certain garbage collection scenarios
* Extension function calls in XSLT variable declarations could break the
stylesheet and crash on repeated calls
* Deep copying Elements could lose namespaces declared in parents
* Deep copying Elements did not copy tail
* Parsing file(-like) objects failed to load external entities
* Parsing 8-bit strings from file(-like) objects raised an exception
* xsl:include failed when the stylesheet was parsed from a file-like object
* lxml.sax.ElementTreeProducer did not call startDocument() / endDocument()
* MSVC compiler complained about long strings (supports only 2048 bytes)
1.0 (2006-06-01)
================
Features added
--------------
* Element.getiterator() and the findall() methods support finding arbitrary
elements from a namespace (pattern ``{namespace}*``)
* Another speedup in tree iteration code
* General speedup of Python Element object creation and deallocation
* Writing C14N no longer serializes in memory (reduced memory footprint)
* PyErrorLog for error logging through the Python ``logging`` module
* ``Element.getroottree()`` returns an ElementTree for the root node of the
document that contains the element.
* ElementTree.getpath(element) returns a simple, absolute XPath expression to
find the element in the tree structure
* Error logs have a ``last_error`` attribute for convenience
* Comment texts can be changed through the API
* Formatted output via ``pretty_print`` keyword in serialization functions
* XSLT can block access to file system and network via ``XSLTAccessControl``
* ElementTree.write() no longer serializes in memory (reduced memory
footprint)
* Speedup of Element.findall(tag) and Element.getiterator(tag)
* Support for writing the XML representation of Elements and ElementTrees to
Python unicode strings via ``etree.tounicode()``
* Support for writing XSLT results to Python unicode strings via ``unicode()``
* Parsing a unicode string no longer copies the string (reduced memory
footprint)
* Parsing file-like objects reads chunks rather than the whole file (reduced
memory footprint)
* Parsing StringIO objects from the start avoids copying the string (reduced
memory footprint)
* Read-only 'docinfo' attribute in ElementTree class holds DOCTYPE
information, original encoding and XML version as seen by the parser
* etree module can be compiled without libxslt by commenting out the line
``include "xslt.pxi"`` near the end of the etree.pyx source file
* Better error messages in parser exceptions
* Error reporting also works in XSLT
* Support for custom document loaders (URI resolvers) in parsers and XSLT,
resolvers are registered at parser level
* Implementation of exslt:regexp for XSLT based on the Python 're' module,
enabled by default, can be switched off with 'regexp=False' keyword argument
* Support for exslt extensions (libexslt) and libxslt extra functions
(node-set, document, write, output)
* Substantial speedup in XPath.evaluate()
* HTMLParser for parsing (broken) HTML
* XMLDTDID function parses XML into tuple (root node, ID dict) based on xml:id
implementation of libxml2 (as opposed to ET compatible XMLID)
Bugs fixed
----------
* Memory leak in Element.__setitem__
* Memory leak in Element.attrib.items() and Element.attrib.values()
* Memory leak in XPath extension functions
* Memory leak in unicode related setup code
* Element now raises ValueError on empty tag names
* Namespace fixing after moving elements between documents could fail if the
source document was freed too early
* Setting namespace-less tag names on namespaced elements ('{ns}t' -> 't')
didn't reset the namespace
* Unknown constants from newer libxml2 versions could raise exceptions in the
error handlers
* lxml.etree compiles much faster
* On libxml2 <= 2.6.22, parsing strings with encoding declaration could fail
in certain cases
* Document reference in ElementTree objects was not updated when the root
element was moved to a different document
* Running absolute XPath expressions on an Element now evaluates against the
root tree
* Evaluating absolute XPath expressions (``/*``) on an ElementTree could fail
* Crashes when calling XSLT, RelaxNG, etc. with uninitialized ElementTree
objects
* Removed public function ``initThreadLogging()``, replaced by more general
``initThread()`` which fixes a number of setup problems in threads
* Memory leak when using iconv encoders in tostring/write
* Deep copying Elements and ElementTrees maintains the document information
* Serialization functions raise LookupError for unknown encodings
* Memory deallocation crash resulting from deep copying elements
* Some ElementTree methods could crash if the root node was not initialized
(neither file nor element passed to the constructor)
* Element/SubElement failed to set attribute namespaces from passed ``attrib``
dictionary
* ``tostring()`` adds an XML declaration for non-ASCII encodings
* ``tostring()`` failed to serialize encodings that contain 0-bytes
* ElementTree.xpath() and XPathDocumentEvaluator were not using the
ElementTree root node as reference point
* Calling ``document('')`` in XSLT failed to return the stylesheet
0.9.2 (2006-05-10)
==================
Features added
--------------
* Speedup for Element.makeelement(): the new element reuses the original
libxml2 document instead of creating a new empty one
* Speedup for reversed() iteration over element children (Py2.4+ only)
* ElementTree compatible QName class
* RelaxNG and XMLSchema accept any Element, not only ElementTrees
Bugs fixed
----------
* str(xslt_result) was broken for XSLT output other than UTF-8
* Memory leak if write_c14n fails to write the file after conversion
* Crash in XMLSchema and RelaxNG when passing non-schema documents
* Memory leak in RelaxNG() when RelaxNGParseError is raised
0.9.1 (2006-03-30)
==================
Features added
--------------
* lxml.sax.ElementTreeContentHandler checks closing elements and raises
SaxError on mismatch
* lxml.sax.ElementTreeContentHandler supports namespace-less SAX events
(startElement, endElement) and defaults to empty attributes (keyword
argument)
* Speedup for repeatedly accessing element tag names
* Minor API performance improvements
Bugs fixed
----------
* Memory deallocation bug when using XSLT output method "html"
* sax.py was handling UTF-8 encoded tag names where it shouldn't
* lxml.tests package will no longer be installed (is still in source tar)
0.9 (2006-03-20)
================
Features added
--------------
* Error logging API for libxml2 error messages
* Various performance improvements
* Benchmark script for lxml, ElementTree and cElementTree
* Support for registering extension functions through new FunctionNamespace
class (see doc/extensions.txt)
* ETXPath class for XPath expressions in ElementTree notation ('//{ns}tag')
* Support for variables in XPath expressions (also in XPath class)
* XPath class for compiled XPath expressions
* XMLID module level function (ElementTree compatible)
* XMLParser API for customized libxml2 parser configuration
* Support for custom Element classes through new Namespace API (see
doc/namespace_extensions.txt)
* Common exception base class LxmlError for module exceptions
* real iterator support in iter(Element), Element.getiterator()
* XSLT objects are callable, result trees support str()
* Added MANIFEST.in for easier creation of RPM files.
* 'getparent' method on elements allows navigation to an element's
parent element.
* Python core compatible SAX tree builder and SAX event generator. See
doc/sax.txt for more information.
Bugs fixed
----------
* Segfaults and memory leaks in various API functions of Element
* Segfault in XSLT.tostring()
* ElementTree objects no longer interfere, Elements can be root of different
ElementTrees at the same time
* document('') works in XSLT documents read from files (in-memory documents
cannot support this due to libxslt deficiencies)
0.8 (2005-11-03)
================
Features added
--------------
* Support for copy.deepcopy() on elements. copy.copy() works also, but
does the same thing, and does *not* create a shallow copy, as that
makes no sense in the context of libxml2 trees. This means a
potential incompatibility with ElementTree, but there's more chance
that it works than if copy.copy() isn't supported at all.
* Increased compatibility with (c)ElementTree; .parse() on ElementTree is
supported and parsing of gzipped XML files works.
* implemented index() on elements, allowing one to find the index of a
SubElement.
Bugs fixed
----------
* Use xslt-config instead of xml2-config to find out libxml2
directories to take into account a case where libxslt is installed
in a different directory than libxslt.
* Eliminate crash condition in iteration when text nodes are changed.
* Passing 'None' to tostring() does not result in a segfault anymore,
but an AssertionError.
* Some test fixes for Windows.
* Raise XMLSyntaxError and XPathSyntaxError instead of plain python
syntax errors. This should be less confusing.
* Fixed error with uncaught exception in Pyrex code.
* Calling lxml.etree.fromstring('') throws XMLSyntaxError instead of a
segfault.
* has_key() works on attrib. 'in' tests also work correctly on attrib.
* INSTALL.txt was saying 2.2.16 instead of 2.6.16 as a supported
libxml2 version, as it should.
* Passing a UTF-8 encoded string to the XML() function would fail;
fixed.
0.7 (2005-06-15)
================
Features added
--------------
* parameters (XPath expressions) can be passed to XSLT using keyword
parameters.
* Simple XInclude support. Calling the xinclude() method on a tree
will process any XInclude statements in the document.
* XMLSchema support. Use the XMLSchema class or the convenience
xmlschema() method on a tree to do XML Schema (XSD) validation.
* Added convenience xslt() method on tree. This is less efficient
than the XSLT object, but makes it easier to write quick code.
* Added convenience relaxng() method on tree. This is less efficient
than the RelaxNG object, but makes it easier to write quick code.
* Make it possible to use XPathEvaluator with elements as well. The
XPathEvaluator in this case will retain the element so multiple
XPath queries can be made against one element efficiently. This
replaces the second argument to the .evaluate() method that existed
previously.
* Allow registerNamespace() to be called on an XPathEvaluator, after
creation, to add additional namespaces. Also allow registerNamespaces(),
which does the same for a namespace dictionary.
* Add 'prefix' attribute to element to be able to read prefix information.
This is entirely read-only.
* It is possible to supply an extra nsmap keyword parameter to
the Element() and SubElement() constructors, which supplies a
prefix to namespace URI mapping. This will create namespace
prefix declarations on these elements and these prefixes will show up
in XML serialization.
Bugs fixed
----------
* Killed yet another memory management related bug: trees created
using newDoc would not get a libxml2-level dictionary, which caused
problems when deallocating these documents later if they contained a
node that came from a document with a dictionary.
* Moving namespaced elements between documents was problematic as
references to the original document would remain. This has been fixed
by applying xmlReconciliateNs() after each move operation.
* Can pass None to 'dump()' without segfaults.
* tostring() works properly for non-root elements as well.
* Cleaned out the tostring() method so it should handle encoding
correctly.
* Cleaned out the ElementTree.write() method so it should handle encoding
correctly. Writing directly to a file should also be faster, as there is no
need to go through a Python string in that case. Made sure the test cases
test both serializing to StringIO as well as serializing to a real file.
0.6 (2005-05-14)
================
Features added
--------------
* Changed setup.py so that library_dirs is also guessed. This should
help with compilation on the Mac OS X platform, where otherwise the
wrong library (shipping with the OS) could be picked up.
* Tweaked setup.py so that it picks up the version from version.txt.
Bugs fixed
----------
* Do the right thing when handling namespaced attributes.
* fix bug where tostring() moved nodes into new documents. tostring()
had very nasty side-effects before this fix, sorry!
0.5.1 (2005-04-09)
==================
* Python 2.2 compatibility fixes.
* unicode fixes in Element() and Comment() as well as XML(); unicode
input wasn't properly being UTF-8 encoded.
0.5 (2005-04-08)
================
Initial public release.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/CREDITS.txt 0000644 0001751 0000200 00000003646 15001705132 013725 0 ustar 00runner docker =======
Credits
=======
Main contributors
=================
Stefan Behnel
main developer and maintainer
Martijn Faassen
creator of lxml and initial main developer
Ian Bicking
creator and maintainer of lxml.html
Holger Joukl
ISO-Schematron support, development on lxml.objectify, bug reports, feedback
Simon Sapin
external maintenance and development of the cssselect package
Marc-Antoine Parent
XPath extension function help and patches
Olivier Grisel
improved (c)ElementTree compatibility patches,
website improvements.
Kasimier Buchcik
help with specs and libxml2
Florian Wagner
help with copy.deepcopy support, bug reporting
Emil Kroymann
help with encoding support, bug reporting
Paul Everitt
bug reporting, feedback on API design
Victor Ng
Discussions on memory management strategies, vlibxml2
Robert Kern
feedback on API design
Andreas Pakulat
rpath linking support, doc improvements
David Sankel
building statically on Windows
Marcin Kasperski
PDF documentation generation
Sidnei da Silva
official MS Windows builds
Pascal Oberndörfer
official Mac-OS builds
... and lots of other people who contributed to lxml by reporting
bugs, discussing its functionality or blaming the docs for the bugs in
their code. Thank you all, user feedback and discussions form a very
important part of an Open Source project!
Special thanks goes to:
=======================
* Daniel Veillard and the libxml2 project for a great XML library.
* Fredrik Lundh for ElementTree, its API, and the competition through
cElementTree.
* Greg Ewing (Pyrex) and Robert Bradshaw et al. (Cython) for the
binding technology.
* Jonathan Stoppani for hosting the new mailing list on lxml.de.
* the codespeak crew, in particular Philipp von Weitershausen and
Holger Krekel for originally hosting lxml on codespeak.net
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/INSTALL.txt 0000644 0001751 0000200 00000017073 15001705132 013735 0 ustar 00runner docker Installing lxml
===============
.. contents:: :depth: 1
..
1 Where to get it
2 Requirements
3 Installation
4 Building lxml from dev sources
5 Using lxml with python-libxml2
6 Source builds on MS Windows
7 Source builds on MacOS-X
Where to get it
---------------
lxml is generally distributed through PyPI_.
.. _PyPI: http://pypi.python.org/pypi/lxml
Most **Linux** platforms come with some version of lxml readily
packaged, usually named ``python-lxml`` for the Python 2.x version
and ``python3-lxml`` for Python 3.x. If you can use that version,
the quickest way to install lxml is to use the system package
manager, e.g. ``apt-get`` on Debian/Ubuntu::
sudo apt-get install python3-lxml
For **MacOS-X**, a `macport `_) and
run the following::
pip install lxml
If you are not using pip in a virtualenv and want to install lxml globally
instead, you have to run the above command as admin, e.g. on Linux::
sudo pip install lxml
To install a specific version, either download the distribution
manually and let pip install that, or pass the desired version
to pip::
pip install lxml==5.0.0
.. _pip: http://pypi.python.org/pypi/pip
To speed up the build in test environments, e.g. on a continuous
integration server, disable the C compiler optimisations by setting
the ``CFLAGS`` environment variable::
CFLAGS="-O0" pip install lxml
(The option reads "minus Oh Zero", i.e. zero optimisations.)
MS Windows
..........
For MS Windows, we try to provide binary wheels with reasonably up-to-date
libraries, although you might still want to take a look at the related
`FAQ entry `_.
Since it is generally difficult to build software on Windows, the library
versions (libxml2, libxslt, libiconv, zlib) might not always be at the
same version level as the builds on Linux or macOS. This usually means
that the `WinLibs project `_
has not updated their repositories yet. If you need a more recent version,
please file a ticket on their side to update it.
Linux
.....
On Linux (and most other well-behaved operating systems), ``pip`` will
manage to build the source distribution as long as libxml2 and libxslt
are properly installed, including development packages, i.e. header files,
etc. See the requirements section above and use your system package
management tool to look for packages like ``libxml2-dev`` or
``libxslt-devel``. If the build fails, make sure they are installed.
Alternatively, setting ``STATIC_DEPS=true`` will download and build
both libraries automatically in their latest version, e.g.
``STATIC_DEPS=true pip install lxml``.
MacOS-X
.......
On MacOS-X, we provide binary wheels ("universal2" for Python 3.9+),
so just use::
sudo pip3 install lxml
To build the source distribution, use the following and
make sure you have a working Internet connection, as this will
download libxml2 and libxslt in order to build them::
STATIC_DEPS=true sudo pip install lxml
Building lxml from dev sources
------------------------------
If you want to build lxml from the GitHub repository, you should read
`how to build lxml from source`_ (or the file ``doc/build.txt`` in the
source tree). Building from developer sources or from modified
distribution sources requires Cython_ to translate the lxml sources
into C code. The source distribution ships with pre-generated C
source files, so you do not need Cython installed to build from
release sources.
.. _Cython: http://www.cython.org
.. _`how to build lxml from source`: build.html
If you have read these instructions and still cannot manage to install lxml,
you can check the archives of the `mailing list`_ to see if your problem is
known or otherwise send a mail to the list.
.. _`mailing list`: http://lxml.de/mailinglist/
Using lxml with python-libxml2
------------------------------
If you want to use lxml together with the official libxml2 Python
bindings (maybe because one of your dependencies uses it), you must
build lxml statically. Otherwise, the two packages will interfere in
places where the libxml2 library requires global configuration, which
can have any kind of effect from disappearing functionality to crashes
in either of the two.
To get a static build, either pass the ``--static-deps`` option to the
setup.py script, or run ``pip`` with the ``STATIC_DEPS`` or
``STATICBUILD`` environment variable set to true, i.e.
::
STATIC_DEPS=true pip install lxml
The ``STATICBUILD`` environment variable is handled equivalently to
the ``STATIC_DEPS`` variable, but is used by some other extension
packages, too.
Source builds on MS Windows
---------------------------
Most MS Windows systems lack the necessarily tools to build software,
starting with a C compiler already. Microsoft leaves it to users to
install and configure them, which is usually not trivial and means
that distributors cannot rely on these dependencies being available
on a given system. In a way, you get what you've paid for and make
others pay for it.
Due to the additional lack of package management of this platform,
it is best to link the library dependencies statically if you decide
to build from sources, rather than using a binary installer. For
that, lxml can use the `binary distribution of libxml2 and libxslt
`_, which it downloads
automatically during the static build. It needs both libxml2 and
libxslt, as well as iconv and zlib, which are available from the
same download site. Further build instructions are in the
`source build documentation `_.
Source builds on MacOS-X
------------------------
If you are not using macports or want to use a more recent lxml
release, you have to build it yourself. While the pre-installed system
libraries of libxml2 and libxslt are less outdated in recent MacOS-X
versions than they used to be, so lxml should work with them out of the
box, it is still recommended to use a static build with the most recent
library versions.
Luckily, lxml's ``setup.py`` script has built-in support for building
and integrating these libraries statically during the build. Please
read the
`MacOS-X build instructions `_.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/LICENSE.txt 0000644 0001751 0000200 00000002720 15001705132 013702 0 ustar 00runner docker Copyright (c) 2004 Infrae. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
3. Neither the name of Infrae nor the names of its contributors may
be used to endorse or promote products derived from this software
without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INFRAE OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/LICENSES.txt 0000644 0001751 0000200 00000002752 15001705132 014032 0 ustar 00runner docker lxml is copyright Infrae and distributed under the BSD license (see
doc/licenses/BSD.txt), with the following exceptions:
Some code, such a selftest.py, selftest2.py and
src/lxml/_elementpath.py are derived from ElementTree and
cElementTree. See doc/licenses/elementtree.txt for the license text.
lxml.cssselect and lxml.html are copyright Ian Bicking and distributed
under the BSD license (see doc/licenses/BSD.txt).
test.py, the test-runner script, is GPL and copyright Shuttleworth
Foundation. See doc/licenses/GPL.txt. It is believed the unchanged
inclusion of test.py to run the unit test suite falls under the
"aggregation" clause of the GPL and thus does not affect the license
of the rest of the package.
The isoschematron implementation uses several XSL and RelaxNG resources:
* The (XML syntax) RelaxNG schema for schematron, copyright International
Organization for Standardization (see
src/lxml/isoschematron/resources/rng/iso-schematron.rng for the license
text)
* The skeleton iso-schematron-xlt1 pure-xslt schematron implementation
xsl stylesheets, copyright Rick Jelliffe and Academia Sinica Computing
Center, Taiwan (see the xsl files here for the license text:
src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/)
* The xsd/rng schema schematron extraction xsl transformations are unlicensed
and copyright the respective authors as noted (see
src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl and
src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/MANIFEST.in 0000644 0001751 0000200 00000001736 15001705132 013623 0 ustar 00runner docker exclude *.py
include setup.py setupinfo.py versioninfo.py buildlibxml.py
include test.py
include update-error-constants.py
include MANIFEST.in Makefile requirements.txt
include CHANGES.txt CREDITS.txt INSTALL.txt LICENSES.txt README.rst TODO.txt
include tools/*.py tools/manylinux/*.sh
include src/lxml/*.c src/lxml/html/*.c
include doc/html/*.png
recursive-include src *.pyx *.pxd *.pxi *.py
recursive-include src/lxml lxml.etree.h lxml.etree_api.h etree.h etree_api.h etree_defs.h lxml_endian.h
recursive-include src/lxml/isoschematron *.rng *.xsl *.txt
recursive-include src/lxml/tests *.rng *.rnc *.xslt *.xml *.dtd *.xsd *.sch *.html *.txt
recursive-include src/lxml/html/tests *.data *.txt
recursive-include samples *.xml
recursive-include benchmark *.py
recursive-include doc *.py *.txt *.html *.css *.xml *.mgp pubkey.asc Makefile
recursive-include doc/html/apidoc *.js *.png *.inv
recursive-include doc/s5/ui *.gif *.htc *.png *.js
recursive-include doc/s5/ep2008 *.py *.png *.rng
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/Makefile 0000644 0001751 0000200 00000013127 15001705132 013522 0 ustar 00runner docker PYTHON?=python3
TESTFLAGS=-p -vv
TESTOPTS=
SETUPFLAGS=
LXMLVERSION:=$(shell $(PYTHON) -c 'import re; print(re.findall(r"__version__\s*=\s*\"([^\"]+)\"", open("src/lxml/__init__.py").read())[0])' )
PYTHON_WITH_CYTHON?=$(shell $(PYTHON) -c 'import Cython.Build.Dependencies' >/dev/null 2>/dev/null && echo " --with-cython" || true)
CYTHON_WITH_COVERAGE?=$(shell $(PYTHON) -c 'import Cython.Coverage; import sys; assert not hasattr(sys, "pypy_version_info")' >/dev/null 2>/dev/null && echo " --coverage" || true)
PYTHON_BUILD_VERSION ?= *
MANYLINUX_LIBXML2_VERSION=2.13.8
MANYLINUX_LIBXSLT_VERSION=1.1.43
MANYLINUX_CFLAGS=-O3 -g1 -pipe -fPIC -flto
MANYLINUX_LDFLAGS=-flto
MANYLINUX_IMAGES= \
manylinux1_x86_64 \
manylinux1_i686 \
manylinux_2_24_x86_64 \
manylinux_2_24_i686 \
manylinux2014_aarch64 \
manylinux_2_24_aarch64 \
manylinux_2_24_ppc64le \
manylinux_2_28_x86_64 \
manylinux_2_28_aarch64 \
manylinux_2_28_ppc64le \
manylinux_2_24_s390x \
musllinux_1_1_x86_64 \
musllinux_1_1_aarch64
.PHONY: all inplace rebuild-sdist sdist build require-cython wheel_manylinux wheel
all: inplace
# Build in-place
inplace:
$(PYTHON) setup.py $(SETUPFLAGS) build_ext -i $(PYTHON_WITH_CYTHON) --warnings $(subst --,--with-,$(CYTHON_WITH_COVERAGE)) -j7
rebuild-sdist: require-cython
rm -f dist/lxml-$(LXMLVERSION).tar.gz
find src -name '*.c' -exec rm -f {} \;
$(MAKE) dist/lxml-$(LXMLVERSION).tar.gz
dist/lxml-$(LXMLVERSION).tar.gz:
$(PYTHON) setup.py $(SETUPFLAGS) sdist $(PYTHON_WITH_CYTHON)
sdist: dist/lxml-$(LXMLVERSION).tar.gz
build:
$(PYTHON) setup.py $(SETUPFLAGS) build $(PYTHON_WITH_CYTHON) --warnings
require-cython:
@[ -n "$(PYTHON_WITH_CYTHON)" ] || { \
echo "NOTE: missing Cython - please use this command to install it: $(PYTHON) -m pip install Cython"; false; }
qemu-user-static:
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
wheel_manylinux: $(addprefix wheel_,$(MANYLINUX_IMAGES))
$(addprefix wheel_,$(filter-out %_x86_64, $(filter-out %_i686, $(MANYLINUX_IMAGES)))): qemu-user-static
wheel_%: dist/lxml-$(LXMLVERSION).tar.gz
time docker run --rm -t \
-v $(shell pwd):/io \
-e AR=gcc-ar \
-e NM=gcc-nm \
-e RANLIB=gcc-ranlib \
-e CFLAGS="$(MANYLINUX_CFLAGS) $(if $(patsubst %aarch64,,$@),-march=core2,-march=armv8-a -mtune=cortex-a72)" \
-e LDFLAGS="$(MANYLINUX_LDFLAGS)" \
-e STATIC_DEPS="${STATIC_DEPS}" \
-e LIBXML2_VERSION="$(MANYLINUX_LIBXML2_VERSION)" \
-e LIBXSLT_VERSION="$(MANYLINUX_LIBXSLT_VERSION)" \
-e PYTHON_BUILD_VERSION="$(PYTHON_BUILD_VERSION)" \
-e WHEELHOUSE=$(subst wheel_,wheelhouse/,$@) \
quay.io/pypa/$(subst wheel_,,$@) \
bash /io/tools/manylinux/build-wheels.sh /io/$<
wheel:
$(PYTHON) setup.py $(SETUPFLAGS) bdist_wheel $(PYTHON_WITH_CYTHON) --warnings
wheel_static:
$(PYTHON) setup.py $(SETUPFLAGS) bdist_wheel $(PYTHON_WITH_CYTHON) --warnings --static-deps
test_build: build
$(PYTHON) test.py $(TESTFLAGS) $(TESTOPTS)
test_inplace: inplace
$(PYTHON) test.py $(TESTFLAGS) $(TESTOPTS) $(CYTHON_WITH_COVERAGE)
valgrind_test_inplace: inplace
valgrind --tool=memcheck --leak-check=full --num-callers=30 --suppressions=valgrind-python.supp \
$(PYTHON) test.py
fuzz: clean
$(MAKE) \
CC="/usr/bin/clang" \
CFLAGS="$$CFLAGS -fsanitize=fuzzer-no-link -g2" \
CXX="/usr/bin/clang++" \
CXXFLAGS="-fsanitize=fuzzer-no-link" \
inplace
$(PYTHON) src/lxml/tests/fuzz_xml_parse.py
gdb_test_inplace: inplace
@echo "file $(PYTHON)\nrun test.py" > .gdb.command
gdb -x .gdb.command -d src -d src/lxml
bench_inplace: inplace
$(PYTHON) benchmark/bench_etree.py -i
$(PYTHON) benchmark/bench_xpath.py -i
$(PYTHON) benchmark/bench_xslt.py -i
$(PYTHON) benchmark/bench_objectify.py -i
ftest_build: build
$(PYTHON) test.py -f $(TESTFLAGS) $(TESTOPTS)
ftest_inplace: inplace
$(PYTHON) test.py -f $(TESTFLAGS) $(TESTOPTS)
apidoc: apidocclean inplace
@[ -x "`command -v sphinx-apidoc`" ] \
&& (echo "Generating API docs ..." && \
PYTHONPATH=src:$(PYTHONPATH) sphinx-apidoc -e -P -T -o doc/api src/lxml \
"*includes" "*tests" "*pyclasslookup.py" "*usedoctest.py" "*html/_html5builder.py" \
"*.so" "*.pyd") \
|| (echo "not generating Sphinx autodoc API rst files")
apihtml: apidoc inplace
@[ -x "`command -v sphinx-build`" ] \
&& (echo "Generating API docs ..." && \
make -C doc/api html) \
|| (echo "not generating Sphinx autodoc API documentation")
website: inplace docclean
PYTHONPATH=src:$(PYTHONPATH) $(PYTHON) doc/mkhtml.py doc/html . ${LXMLVERSION}
html: apihtml website s5
s5:
$(MAKE) -C doc/s5 slides
apipdf: apidoc inplace
rm -fr doc/api/_build
@[ -x "`command -v sphinx-build`" ] \
&& (echo "Generating API PDF docs ..." && \
make -C doc/api latexpdf) \
|| (echo "not generating Sphinx autodoc API PDF documentation")
pdf: apipdf pdfclean
$(PYTHON) doc/mklatex.py doc/pdf . ${LXMLVERSION}
(cd doc/pdf && pdflatex lxmldoc.tex \
&& pdflatex lxmldoc.tex \
&& pdflatex lxmldoc.tex)
@cp doc/pdf/lxmldoc.pdf doc/pdf/lxmldoc-${LXMLVERSION}.pdf
@echo "PDF available as doc/pdf/lxmldoc-${LXMLVERSION}.pdf"
# Two pdflatex runs are needed to build the correct Table of contents.
test: test_inplace
valtest: valgrind_test_inplace
gdbtest: gdb_test_inplace
bench: bench_inplace
ftest: ftest_inplace
clean:
find src \( -name '*.o' -o -name '*.so' -o -name '*.py[cod]' -o -name '*.dll' \) -exec rm -f {} \;
rm -rf build
docclean:
$(MAKE) -C doc/s5 clean
rm -f doc/html/*.html
pdfclean:
rm -fr doc/pdf
apidocclean:
rm -fr doc/html/api
rm -f doc/api/lxml*.rst
rm -fr doc/api/_build
realclean: clean docclean apidocclean
find src -name '*.c' -exec rm -f {} \;
rm -f TAGS
$(PYTHON) setup.py clean -a --without-cython
././@PaxHeader 0000000 0000000 0000000 00000000034 00000000000 010212 x ustar 00 28 mtime=1745324802.7136004
lxml-5.4.0/PKG-INFO 0000644 0001751 0000200 00000006672 15001705403 013167 0 ustar 00runner docker Metadata-Version: 2.4
Name: lxml
Version: 5.4.0
Summary: Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API.
Home-page: https://lxml.de/
Author: lxml dev team
Author-email: lxml@lxml.de
Maintainer: lxml dev team
Maintainer-email: lxml@lxml.de
License: BSD-3-Clause
Project-URL: Source, https://github.com/lxml/lxml
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Cython
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: C
Classifier: Operating System :: OS Independent
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Text Processing :: Markup :: XML
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.6
License-File: LICENSE.txt
License-File: LICENSES.txt
Provides-Extra: source
Requires-Dist: Cython<3.1.0,>=3.0.11; extra == "source"
Provides-Extra: cssselect
Requires-Dist: cssselect>=0.7; extra == "cssselect"
Provides-Extra: html5
Requires-Dist: html5lib; extra == "html5"
Provides-Extra: htmlsoup
Requires-Dist: BeautifulSoup4; extra == "htmlsoup"
Provides-Extra: html-clean
Requires-Dist: lxml_html_clean; extra == "html-clean"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: maintainer
Dynamic: maintainer-email
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-python
Dynamic: summary
lxml is a Pythonic, mature binding for the libxml2 and libxslt libraries. It
provides safe and convenient access to these libraries using the ElementTree
API.
It extends the ElementTree API significantly to offer support for XPath,
RelaxNG, XML Schema, XSLT, C14N and much more.
To contact the project, go to the `project home page
`_
via a `Tidelift subscription `_
or via PayPal:
|Donate|_
.. _`Donate`: https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=R56JE3VCPDA9N
Please `contact Stefan Behnel `_
supports the lxml project with their build and CI servers.
Project income report
---------------------
lxml has `about 80 million downloads `_
per month on PyPI.
* Total project income in 2023: EUR 2776.56 (231.38 € / month, 2.89 € / 1,000,000 downloads)
- Tidelift: EUR 2738.46
- Paypal: EUR 38.10
* Total project income in 2022: EUR 2566.38 (213.87 € / month, 3.56 € / 1,000,000 downloads)
- Tidelift: EUR 2539.38
- Paypal: EUR 24.32
* Total project income in 2021: EUR 4640.37 (386.70 € / month)
- Tidelift: EUR 4066.66
- Paypal: EUR 223.71
- other: EUR 350.00
* Total project income in 2020: EUR 6065,86 (506.49 € / month)
- Tidelift: EUR 4064.77
- Paypal: EUR 1401.09
- other: EUR 600.00
* Total project income in 2019: EUR 717.52 (59.79 € / month)
- Tidelift: EUR 360.30
- Paypal: EUR 157.22
- other: EUR 200.00
Legal Notice for Donations
--------------------------
Any donation that you make to the lxml project is voluntary and
is not a fee for any services, goods, or advantages. By making
a donation to the lxml project, you acknowledge that we have the
right to use the money you donate in any lawful way and for any
lawful purpose we see fit and we are not obligated to disclose
the way and purpose to any party unless required by applicable
law. Although lxml is free software, to the best of our knowledge
the lxml project does not have any tax exempt status. The lxml
project is neither a registered non-profit corporation nor a
registered charity in any country. Your donation may or may not
be tax-deductible; please consult your tax advisor in this matter.
We will not publish or disclose your name and/or e-mail address
without your consent, unless required by applicable law. Your
donation is non-refundable.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/TODO.txt 0000644 0001751 0000200 00000002663 15001705132 013373 0 ustar 00runner docker ===============
ToDo's for lxml
===============
lxml
====
In general
----------
* more testing on multi-threading
* better exception messages for XPath and schemas based on error log,
e.g. missing namespace mappings in XPath
* when building statically, compile everything into one shared library
instead of one for lxml.etree and one for lxml.objectify to prevent
the redundant static linking of the library dependencies.
* more testing on input/output of encoded filenames, including custom
resolvers, relative XSLT imports, ...
* always use '' as URL when tree was parsed from string? (can libxml2
handle this?)
* follow PEP 8 in API naming (avoidCamelCase in_favour_of_underscores)
* use per-call or per-thread error logs in XSLT/XPath/etc. to keep the
messages separate, especially in exceptions
* add 'nsmap' parameter to cleanup_namespaces()
* fix tail text handling in addnext()/addprevious()
* make Element nsmap editable to allow defining new namespaces (LP#555602)
Entities
--------
* clean support for entities (is the Entity element class enough?)
Objectify
---------
* emulate setting special __attributes__ on ObjectifiedElement's as Python
attributes, not XML children
Incremental parsing
-------------------
* create all iterparse events only on start events and
store the end events in the stack
* rewrite SAX event creation in a more C-ish way to avoid having to
acquire the GIL on each event
././@PaxHeader 0000000 0000000 0000000 00000000031 00000000000 010207 x ustar 00 25 mtime=1745324802.6166
lxml-5.4.0/benchmark/ 0000755 0001751 0000200 00000000000 15001705403 014011 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/benchmark/bench_etree.py 0000644 0001751 0000200 00000027604 15001705132 016636 0 ustar 00runner docker import copy
from io import BytesIO
from itertools import *
import benchbase
from benchbase import (with_attributes, with_text, onlylib,
serialized, children, nochange)
TEXT = "some ASCII text"
UTEXT = u"some klingon: \uF8D2"
############################################################
# Benchmarks
############################################################
class BenchMark(benchbase.TreeBenchMark):
@nochange
def bench_iter_children(self, root):
for child in root:
pass
@nochange
def bench_iter_children_reversed(self, root):
for child in reversed(root):
pass
@nochange
def bench_first_child(self, root):
for i in self.repeat1000:
child = root[0]
@nochange
def bench_last_child(self, root):
for i in self.repeat1000:
child = root[-1]
@nochange
def bench_middle_child(self, root):
pos = len(root) // 2
for i in self.repeat1000:
child = root[pos]
@nochange
@with_attributes(False)
@with_text(text=True)
def bench_tostring_text_ascii(self, root):
self.etree.tostring(root, method="text")
@nochange
@with_attributes(False)
@with_text(text=True, utext=True)
def bench_tostring_text_unicode(self, root):
self.etree.tostring(root, method="text", encoding='unicode')
@nochange
@with_attributes(False)
@with_text(text=True, utext=True)
def bench_tostring_text_utf16(self, root):
self.etree.tostring(root, method="text", encoding='UTF-16')
@nochange
@with_attributes(False)
@with_text(text=True, utext=True)
@onlylib('lxe')
@children
def bench_tostring_text_utf8_with_tail(self, children):
for child in children:
self.etree.tostring(child, method="text",
encoding='UTF-8', with_tail=True)
@nochange
@with_attributes(True, False)
@with_text(text=True, utext=True)
def bench_tostring_utf8(self, root):
self.etree.tostring(root, encoding='UTF-8')
@nochange
@with_attributes(True, False)
@with_text(text=True, utext=True)
def bench_tostring_utf16(self, root):
self.etree.tostring(root, encoding='UTF-16')
@nochange
@with_attributes(True, False)
@with_text(text=True, utext=True)
def bench_tostring_utf8_unicode_XML(self, root):
xml = self.etree.tostring(root, encoding='UTF-8').decode('UTF-8')
self.etree.XML(xml)
@nochange
@with_attributes(True, False)
@with_text(text=True, utext=True)
def bench_write_utf8_parse_bytesIO(self, root):
f = BytesIO()
self.etree.ElementTree(root).write(f, encoding='UTF-8')
f.seek(0)
self.etree.parse(f)
@with_attributes(True, False)
@with_text(text=True, utext=True)
@serialized
def bench_parse_bytesIO(self, root_xml):
f = BytesIO(root_xml)
self.etree.parse(f)
@with_attributes(True, False)
@with_text(text=True, utext=True)
@serialized
def bench_XML(self, root_xml):
self.etree.XML(root_xml)
@with_attributes(True, False)
@with_text(text=True, utext=True)
@serialized
def bench_iterparse_bytesIO(self, root_xml):
f = BytesIO(root_xml)
for event, element in self.etree.iterparse(f):
pass
@with_attributes(True, False)
@with_text(text=True, utext=True)
@serialized
def bench_iterparse_bytesIO_clear(self, root_xml):
f = BytesIO(root_xml)
for event, element in self.etree.iterparse(f):
element.clear()
def bench_append_from_document(self, root1, root2):
# == "1,2 2,3 1,3 3,1 3,2 2,1" # trees 1 and 2, or 2 and 3, or ...
for el in root2:
root1.append(el)
def bench_insert_from_document(self, root1, root2):
pos = len(root1)//2
for el in root2:
root1.insert(pos, el)
pos = pos + 1
def bench_rotate_children(self, root):
# == "1 2 3" # runs on any single tree independently
for i in range(100):
el = root[0]
del root[0]
root.append(el)
def bench_reorder(self, root):
for i in range(1,len(root)//2):
el = root[0]
del root[0]
root[-i:-i] = [ el ]
def bench_reorder_slice(self, root):
for i in range(1,len(root)//2):
els = root[0:1]
del root[0]
root[-i:-i] = els
def bench_clear(self, root):
root.clear()
@nochange
@children
def bench_has_children(self, children):
for child in children:
if child and child and child and child and child:
pass
@nochange
@children
def bench_len(self, children):
for child in children:
map(len, repeat(child, 20))
@children
def bench_create_subelements(self, children):
SubElement = self.etree.SubElement
for child in children:
SubElement(child, '{test}test')
def bench_append_elements(self, root):
Element = self.etree.Element
for child in root:
el = Element('{test}test')
child.append(el)
@nochange
@children
def bench_makeelement(self, children):
empty_attrib = {}
for child in children:
child.makeelement('{test}test', empty_attrib)
@nochange
@children
def bench_create_elements(self, children):
Element = self.etree.Element
for child in children:
Element('{test}test')
@children
def bench_replace_children_element(self, children):
Element = self.etree.Element
for child in children:
el = Element('{test}test')
child[:] = [el]
@children
def bench_replace_children(self, children):
els = [ self.etree.Element("newchild") ]
for child in children:
child[:] = els
def bench_remove_children(self, root):
for child in root:
root.remove(child)
def bench_remove_children_reversed(self, root):
for child in reversed(root):
root.remove(child)
@children
def bench_set_attributes(self, children):
for child in children:
child.set('a', 'bla')
@with_attributes(True)
@children
@nochange
def bench_get_attributes(self, children):
for child in children:
child.get('bla1')
child.get('{attr}test1')
@children
def bench_setget_attributes(self, children):
for child in children:
child.set('a', 'bla')
for child in children:
child.get('a')
@nochange
def bench_root_getchildren(self, root):
root.getchildren()
@nochange
def bench_root_list_children(self, root):
list(root)
@nochange
@children
def bench_getchildren(self, children):
for child in children:
child.getchildren()
@nochange
@children
def bench_get_children_slice(self, children):
for child in children:
child[:]
@nochange
@children
def bench_get_children_slice_2x(self, children):
for child in children:
child[:]
child[:]
@nochange
@children
@with_attributes(True, False)
@with_text(utext=True, text=True, no_text=True)
def bench_deepcopy(self, children):
for child in children:
copy.deepcopy(child)
@nochange
@with_attributes(True, False)
@with_text(utext=True, text=True, no_text=True)
def bench_deepcopy_all(self, root):
copy.deepcopy(root)
@nochange
@children
def bench_tag(self, children):
for child in children:
child.tag
@nochange
@children
def bench_tag_repeat(self, children):
for child in children:
for i in self.repeat100:
child.tag
@nochange
@with_text(utext=True, text=True, no_text=True)
@children
def bench_text(self, children):
for child in children:
child.text
@nochange
@with_text(utext=True, text=True, no_text=True)
@children
def bench_text_repeat(self, children):
for child in children:
for i in self.repeat500:
child.text
@children
def bench_set_text(self, children):
text = TEXT
for child in children:
child.text = text
@children
def bench_set_utext(self, children):
text = UTEXT
for child in children:
child.text = text
@nochange
@onlylib('lxe')
def bench_index(self, root):
for child in root:
root.index(child)
@nochange
@onlylib('lxe')
def bench_index_slice(self, root):
for child in root[5:100]:
root.index(child, 5, 100)
@nochange
@onlylib('lxe')
def bench_index_slice_neg(self, root):
for child in root[-100:-5]:
root.index(child, start=-100, stop=-5)
@nochange
def bench_iter_all(self, root):
list(root.iter())
@nochange
def bench_iter_one_at_a_time(self, root):
list(islice(root.iter(), 2**30, None))
@nochange
def bench_iter_islice(self, root):
list(islice(root.iter(), 10, 110))
@nochange
def bench_iter_tag(self, root):
list(islice(root.iter(self.SEARCH_TAG), 3, 10))
@nochange
def bench_iter_tag_all(self, root):
list(root.iter(self.SEARCH_TAG))
@nochange
def bench_iter_tag_one_at_a_time(self, root):
list(islice(root.iter(self.SEARCH_TAG), 2**30, None))
@nochange
def bench_iter_tag_none(self, root):
list(root.iter("{ThisShould}NeverExist"))
@nochange
def bench_iter_tag_text(self, root):
[ e.text for e in root.iter(self.SEARCH_TAG) ]
@nochange
def bench_findall(self, root):
root.findall(".//*")
@nochange
def bench_findall_child(self, root):
root.findall(".//*/" + self.SEARCH_TAG)
@nochange
def bench_findall_tag(self, root):
root.findall(".//" + self.SEARCH_TAG)
@nochange
def bench_findall_path(self, root):
root.findall(".//*[%s]/./%s/./*" % (self.SEARCH_TAG, self.SEARCH_TAG))
@nochange
@onlylib('lxe')
def bench_xpath_path(self, root):
ns, tag = self.SEARCH_TAG[1:].split('}')
root.xpath(".//*[p:%s]/./p:%s/./*" % (tag,tag),
namespaces = {'p':ns})
@nochange
def bench_iterfind(self, root):
list(root.iterfind(".//*"))
@nochange
def bench_iterfind_tag(self, root):
list(root.iterfind(".//" + self.SEARCH_TAG))
@nochange
def bench_iterfind_islice(self, root):
list(islice(root.iterfind(".//*"), 10, 110))
_bench_xpath_single_xpath = None
@nochange
@onlylib('lxe')
def bench_xpath_single(self, root):
xpath = self._bench_xpath_single_xpath
if xpath is None:
ns, tag = self.SEARCH_TAG[1:].split('}')
xpath = self._bench_xpath_single_xpath = self.etree.XPath(
'.//p:%s[1]' % tag, namespaces={'p': ns})
xpath(root)
@nochange
def bench_find_single(self, root):
root.find(".//%s" % self.SEARCH_TAG)
@nochange
def bench_iter_single(self, root):
next(root.iter(self.SEARCH_TAG))
_bench_xpath_two_xpath = None
@nochange
@onlylib('lxe')
def bench_xpath_two(self, root):
xpath = self._bench_xpath_two_xpath
if xpath is None:
ns, tag = self.SEARCH_TAG[1:].split('}')
xpath = self._bench_xpath_two_xpath = self.etree.XPath(
'.//p:%s[position() < 3]' % tag, namespaces={'p': ns})
xpath(root)
@nochange
def bench_iterfind_two(self, root):
it = root.iterfind(".//%s" % self.SEARCH_TAG)
next(it)
next(it)
@nochange
def bench_iter_two(self, root):
it = root.iter(self.SEARCH_TAG)
next(it)
next(it)
if __name__ == '__main__':
benchbase.main(BenchMark)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/benchmark/bench_objectify.py 0000644 0001751 0000200 00000006254 15001705132 017506 0 ustar 00runner docker from itertools import *
import benchbase
from benchbase import (with_text, children, nochange)
############################################################
# Benchmarks
############################################################
class BenchMark(benchbase.TreeBenchMark):
repeat100 = range(100)
repeat1000 = range(1000)
repeat3000 = range(3000)
def __init__(self, lib):
from lxml import etree, objectify
self.objectify = objectify
parser = etree.XMLParser(remove_blank_text=True)
lookup = objectify.ObjectifyElementClassLookup()
parser.setElementClassLookup(lookup)
super(BenchMark, self).__init__(etree, parser)
@nochange
def bench_attribute(self, root):
"1 2 4"
for i in self.repeat3000:
root.zzzzz
def bench_attribute_assign_int(self, root):
"1 2 4"
for i in self.repeat3000:
root.XYZ = 5
def bench_attribute_assign_string(self, root):
"1 2 4"
for i in self.repeat3000:
root.XYZ = "5"
@nochange
def bench_attribute_cached(self, root):
"1 2 4"
cache = root.zzzzz
for i in self.repeat3000:
root.zzzzz
@nochange
def bench_attributes_deep(self, root):
"1 2 4"
for i in self.repeat3000:
root.zzzzz['{cdefg}a00001']
@nochange
def bench_attributes_deep_cached(self, root):
"1 2 4"
cache1 = root.zzzzz
cache2 = cache1['{cdefg}a00001']
for i in self.repeat3000:
root.zzzzz['{cdefg}a00001']
@nochange
def bench_objectpath(self, root):
"1 2 4"
path = self.objectify.ObjectPath(".zzzzz")
for i in self.repeat3000:
path(root)
@nochange
def bench_objectpath_deep(self, root):
"1 2 4"
path = self.objectify.ObjectPath(".zzzzz.{cdefg}a00001")
for i in self.repeat3000:
path(root)
@nochange
def bench_objectpath_deep_cached(self, root):
"1 2 4"
cache1 = root.zzzzz
cache2 = cache1['{cdefg}a00001']
path = self.objectify.ObjectPath(".zzzzz.{cdefg}a00001")
for i in self.repeat3000:
path(root)
@with_text(text=True, utext=True, no_text=True)
def bench_annotate(self, root):
self.objectify.annotate(root)
@nochange
def bench_descendantpaths(self, root):
root.descendantpaths()
@nochange
@with_text(text=True)
def bench_type_inference(self, root):
"1 2 4"
el = root.aaaaa
for i in self.repeat1000:
el.getchildren()
@nochange
@with_text(text=True)
def bench_type_inference_annotated(self, root):
"1 2 4"
el = root.aaaaa
self.objectify.annotate(el)
for i in self.repeat1000:
el.getchildren()
@nochange
@children
def bench_elementmaker(self, children):
E = self.objectify.E
for child in children:
root = E.this(
"test",
E.will(
E.do("nothing"),
E.special,
)
)
if __name__ == '__main__':
benchbase.main(BenchMark)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/benchmark/bench_xpath.py 0000644 0001751 0000200 00000005154 15001705132 016652 0 ustar 00runner docker from itertools import *
import benchbase
from benchbase import onlylib, children, nochange
############################################################
# Benchmarks
############################################################
class XPathBenchMark(benchbase.TreeBenchMark):
@nochange
@onlylib('lxe')
@children
def bench_xpath_class(self, children):
xpath = self.etree.XPath("./*[1]")
for child in children:
xpath(child)
@nochange
@onlylib('lxe')
@children
def bench_xpath_class_repeat(self, children):
for child in children:
xpath = self.etree.XPath("./*[1]")
xpath(child)
@nochange
@onlylib('lxe')
def bench_xpath_element(self, root):
xpath = self.etree.XPathElementEvaluator(root)
for child in root:
xpath.evaluate("./*[1]")
@nochange
@onlylib('lxe')
@children
def bench_xpath_method(self, children):
for child in children:
child.xpath("./*[1]")
@nochange
@onlylib('lxe')
@children
def bench_multiple_xpath_or(self, children):
xpath = self.etree.XPath(".//p:a00001|.//p:b00001|.//p:c00001",
namespaces={'p':'cdefg'})
for child in children:
xpath(child)
@nochange
@onlylib('lxe')
@children
def bench_multiple_iter_tag(self, children):
for child in children:
list(child.iter("{cdefg}a00001"))
list(child.iter("{cdefg}b00001"))
list(child.iter("{cdefg}c00001"))
@nochange
@onlylib('lxe')
@children
def bench_xpath_old_extensions(self, children):
def return_child(_, elements):
if elements:
return elements[0][0]
else:
return ()
extensions = {("test", "child") : return_child}
xpath = self.etree.XPath("t:child(.)", namespaces={"t":"test"},
extensions=extensions)
for child in children:
xpath(child)
@nochange
@onlylib('lxe')
@children
def bench_xpath_extensions(self, children):
def return_child(_, elements):
if elements:
return elements[0][0]
else:
return ()
self.etree.FunctionNamespace("testns")["t"] = return_child
try:
xpath = self.etree.XPath("test:t(.)", namespaces={"test":"testns"})
for child in children:
xpath(child)
finally:
del self.etree.FunctionNamespace("testns")["t"]
if __name__ == '__main__':
benchbase.main(XPathBenchMark)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/benchmark/bench_xslt.py 0000644 0001751 0000200 00000002753 15001705132 016522 0 ustar 00runner docker from itertools import *
import benchbase
from benchbase import onlylib
############################################################
# Benchmarks
############################################################
class XSLTBenchMark(benchbase.TreeBenchMark):
@onlylib('lxe')
def bench_xslt_extensions_old(self, root):
tree = self.etree.XML("""\
TEST
""")
def return_child(_, elements):
return elements[0][0]
extensions = {('testns', 'child') : return_child}
transform = self.etree.XSLT(tree, extensions)
for i in range(10):
transform(root)
@onlylib('lxe')
def bench_xslt_document(self, root):
transform = self.etree.XSLT(self.etree.XML("""\
TEST
"""))
transform(root)
if __name__ == '__main__':
benchbase.main(XSLTBenchMark)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/benchmark/benchbase.py 0000644 0001751 0000200 00000043044 15001705132 016301 0 ustar 00runner docker import sys, re, string, copy, gc
from itertools import *
import time
try:
izip
except NameError:
izip = zip # Py3
def exec_(code, glob):
if sys.version_info[0] >= 3:
exec(code, glob)
else:
exec("exec code in glob")
TREE_FACTOR = 1 # increase tree size with '-l / '-L' cmd option
_TEXT = "some ASCII text" * TREE_FACTOR
_UTEXT = u"some klingon: \uF8D2" * TREE_FACTOR
_ATTRIBUTES = {
'{attr}test1' : _TEXT,
'{attr}test2' : _TEXT,
'bla1' : _TEXT,
'bla2' : _TEXT,
'bla3' : _TEXT
}
def initArgs(argv):
global TREE_FACTOR
try:
argv.remove('-l')
# use large trees
TREE_FACTOR *= 2
except ValueError:
pass
try:
argv.remove('-L')
# use LARGE trees
TREE_FACTOR *= 2
except ValueError:
pass
############################################################
# benchmark decorators
############################################################
def with_attributes(*use_attributes):
"Decorator for benchmarks that use attributes"
vmap = {False : 0, True : 1}
values = [ vmap[bool(v)] for v in use_attributes ]
def set_value(function):
try:
function.ATTRIBUTES.update(values)
except AttributeError:
function.ATTRIBUTES = set(values)
return function
return set_value
def with_text(no_text=False, text=False, utext=False):
"Decorator for benchmarks that use text"
values = []
if no_text:
values.append(0)
if text:
values.append(1)
if utext:
values.append(2)
def set_value(function):
try:
function.TEXT.add(values)
except AttributeError:
function.TEXT = set(values)
return function
return set_value
def onlylib(*libs):
"Decorator to restrict benchmarks to specific libraries"
def set_libs(function):
if libs:
function.LIBS = libs
return function
return set_libs
def serialized(function):
"Decorator for benchmarks that require serialized XML data"
function.STRING = True
return function
def children(function):
"Decorator for benchmarks that require a list of root children"
function.CHILDREN = True
return function
def nochange(function):
"Decorator for benchmarks that do not change the XML tree"
function.NO_CHANGE = True
return function
############################################################
# benchmark baseclass
############################################################
class SkippedTest(Exception):
pass
class TreeBenchMark(object):
atoz = string.ascii_lowercase
repeat100 = range(100)
repeat500 = range(500)
repeat1000 = range(1000)
_LIB_NAME_MAP = {
'etree' : 'lxe',
'ElementTree' : 'ET',
'cElementTree' : 'cET'
}
SEARCH_TAG = "{cdefg}a00001"
def __init__(self, etree, etree_parser=None):
self.etree = etree
libname = etree.__name__.split('.')[-1]
self.lib_name = self._LIB_NAME_MAP.get(libname, libname)
if libname == 'etree':
deepcopy = copy.deepcopy
def set_property(root, fname):
xml = self._serialize_tree(root)
if etree_parser is not None:
setattr(self, fname, lambda : etree.XML(xml, etree_parser))
else:
setattr(self, fname, lambda : deepcopy(root))
setattr(self, fname + '_xml', lambda : xml)
setattr(self, fname + '_children', lambda : root[:])
else:
def set_property(root, fname):
setattr(self, fname, self.et_make_clone_factory(root))
xml = self._serialize_tree(root)
setattr(self, fname + '_xml', lambda : xml)
setattr(self, fname + '_children', lambda : root[:])
attribute_list = list(enumerate( [{}, _ATTRIBUTES] ))
text_list = list(enumerate( [None, _TEXT, _UTEXT] ))
build_name = self._tree_builder_name
self.setup_times = []
for tree in self._all_trees():
times = []
self.setup_times.append(times)
setup = getattr(self, '_setup_tree%d' % tree)
for an, attributes in attribute_list:
for tn, text in text_list:
root, t = setup(text, attributes)
times.append(t)
set_property(root, build_name(tree, tn, an))
def _tree_builder_name(self, tree, tn, an):
return '_root%d_T%d_A%d' % (tree, tn, an)
def tree_builder(self, tree, tn, an, serial, children):
name = self._tree_builder_name(tree, tn, an)
if serial:
name += '_xml'
elif children:
name += '_children'
return getattr(self, name)
def _serialize_tree(self, root):
return self.etree.tostring(root, encoding='UTF-8')
def et_make_clone_factory(self, elem):
def generate_elem(append, elem, level):
var = "e" + str(level)
arg = repr(elem.tag)
if elem.attrib:
arg += ", **%r" % elem.attrib
if level == 1:
append(" e1 = Element(%s)" % arg)
else:
append(" %s = SubElement(e%d, %s)" % (var, level-1, arg))
if elem.text:
append(" %s.text = %r" % (var, elem.text))
if elem.tail:
append(" %s.tail = %r" % (var, elem.tail))
for e in elem:
generate_elem(append, e, level+1)
# generate code for a function that creates a tree
output = ["def element_factory():"]
generate_elem(output.append, elem, 1)
output.append(" return e1")
# setup global function namespace
namespace = {
"Element" : self.etree.Element,
"SubElement" : self.etree.SubElement
}
# create function object
exec_("\n".join(output), namespace)
return namespace["element_factory"]
def _all_trees(self):
all_trees = []
for name in dir(self):
if name.startswith('_setup_tree'):
all_trees.append(int(name[11:]))
return all_trees
def _setup_tree1(self, text, attributes):
"tree with 26 2nd level and 520 * TREE_FACTOR 3rd level children"
atoz = self.atoz
SubElement = self.etree.SubElement
current_time = time.time
t = current_time()
root = self.etree.Element('{abc}rootnode')
for ch1 in atoz:
el = SubElement(root, "{abc}"+ch1*5, attributes)
el.text = text
for ch2 in atoz:
tag = "{cdefg}%s00001" % ch2
for i in range(20 * TREE_FACTOR):
SubElement(el, tag).tail = text
t = current_time() - t
return root, t
def _setup_tree2(self, text, attributes):
"tree with 520 * TREE_FACTOR 2nd level and 26 3rd level children"
atoz = self.atoz
SubElement = self.etree.SubElement
current_time = time.time
t = current_time()
root = self.etree.Element('{abc}rootnode')
for ch1 in atoz:
for i in range(20 * TREE_FACTOR):
el = SubElement(root, "{abc}"+ch1*5, attributes)
el.text = text
for ch2 in atoz:
SubElement(el, "{cdefg}%s00001" % ch2).tail = text
t = current_time() - t
return root, t
def _setup_tree3(self, text, attributes):
"tree of depth 8 + TREE_FACTOR with 3 children per node"
SubElement = self.etree.SubElement
current_time = time.time
t = current_time()
root = self.etree.Element('{abc}rootnode')
children = [root]
for i in range(6 + TREE_FACTOR):
children = [ SubElement(c, "{cdefg}a%05d" % (i%8), attributes)
for i,c in enumerate(chain(children, children, children)) ]
for child in children:
child.text = text
child.tail = text
t = current_time() - t
return root, t
def _setup_tree4(self, text, attributes):
"small tree with 26 2nd level and 2 3rd level children"
SubElement = self.etree.SubElement
current_time = time.time
t = current_time()
root = self.etree.Element('{abc}rootnode')
for ch1 in self.atoz:
el = SubElement(root, "{abc}"+ch1*5, attributes)
el.text = text
SubElement(el, "{cdefg}a00001", attributes).tail = text
SubElement(el, "{cdefg}z00000", attributes).tail = text
t = current_time() - t
return root, t
def benchmarks(self):
"""Returns a list of all benchmarks.
A benchmark is a tuple containing a method name and a list of tree
numbers. Trees are prepared by the setup function.
"""
all_trees = self._all_trees()
benchmarks = []
for name in dir(self):
if not name.startswith('bench_'):
continue
method = getattr(self, name)
if hasattr(method, 'LIBS') and self.lib_name not in method.LIBS:
method_call = None
else:
method_call = method
if method.__doc__:
tree_sets = method.__doc__.split()
else:
tree_sets = ()
if tree_sets:
tree_tuples = [list(map(int, tree_set.split(',')))
for tree_set in tree_sets]
else:
try:
arg_count = method.func_code.co_argcount - 1
except AttributeError:
try:
arg_count = method.__code__.co_argcount - 1
except AttributeError:
arg_count = 1
tree_tuples = self._permutations(all_trees, arg_count)
serialized = getattr(method, 'STRING', False)
children = getattr(method, 'CHILDREN', False)
no_change = getattr(method, 'NO_CHANGE', False)
for tree_tuple in tree_tuples:
for tn in sorted(getattr(method, 'TEXT', (0,))):
for an in sorted(getattr(method, 'ATTRIBUTES', (0,))):
benchmarks.append((name, method_call, tree_tuple,
tn, an, serialized, children,
no_change))
return benchmarks
def _permutations(self, seq, count):
def _permutations(prefix, remainder, count):
if count == 0:
return [ prefix[:] ]
count -= 1
perms = []
prefix.append(None)
for pos, el in enumerate(remainder):
new_remainder = remainder[:pos] + remainder[pos+1:]
prefix[-1] = el
perms.extend( _permutations(prefix, new_remainder, count) )
prefix.pop()
return perms
return _permutations([], seq, count)
############################################################
# Prepare and run benchmark suites
############################################################
def buildSuites(benchmark_class, etrees, selected):
benchmark_suites = list(map(benchmark_class, etrees))
# sorted by name and tree tuple
benchmarks = [ sorted(b.benchmarks()) for b in benchmark_suites ]
selected = [ re.compile(r).search for r in selected ]
if selected:
benchmarks = [ [ b for b in bs
if [ match for match in selected
if match(b[0]) ] ]
for bs in benchmarks ]
return benchmark_suites, benchmarks
def build_treeset_name(trees, tn, an, serialized, children):
text = {0:'-', 1:'S', 2:'U'}[tn]
attr = {0:'-', 1:'A'}[an]
ser = {True:'X', False:'T'}[serialized]
chd = {True:'C', False:'R'}[children]
return "%s%s%s%s T%s" % (text, attr, ser, chd, ',T'.join(map(str, trees))[:6])
def printSetupTimes(benchmark_suites):
print("Setup times for trees in seconds:")
for b in benchmark_suites:
sys.stdout.write("%-3s: " % b.lib_name)
for an in (0,1):
for tn in (0,1,2):
sys.stdout.write(' %s ' %
build_treeset_name((), tn, an, False, False)[:2])
print('')
for i, tree_times in enumerate(b.setup_times):
print(" T%d: %s" % (i+1, ' '.join("%6.4f" % t for t in tree_times)))
print('')
def runBench(suite, method_name, method_call, tree_set, tn, an,
serial, children, no_change):
if method_call is None:
raise SkippedTest
current_time = time.time
call_repeat = range(10)
tree_builders = [ suite.tree_builder(tree, tn, an, serial, children)
for tree in tree_set ]
rebuild_trees = not no_change and not serial
args = tuple([ build() for build in tree_builders ])
method_call(*args) # run once to skip setup overhead
times = []
for i in range(3):
gc.collect()
gc.disable()
t = -1
for i in call_repeat:
if rebuild_trees:
args = [ build() for build in tree_builders ]
t_one_call = current_time()
method_call(*args)
t_one_call = current_time() - t_one_call
if t < 0:
t = t_one_call
else:
t = min(t, t_one_call)
times.append(1000.0 * t)
gc.enable()
if rebuild_trees:
args = ()
args = ()
gc.collect()
return times
def runBenchmarks(benchmark_suites, benchmarks):
for bench_calls in izip(*benchmarks):
for lib, (bench, benchmark_setup) in enumerate(izip(benchmark_suites, bench_calls)):
bench_name = benchmark_setup[0]
tree_set_name = build_treeset_name(*benchmark_setup[-6:-1])
sys.stdout.write("%-3s: %-28s (%-10s) " % (
bench.lib_name, bench_name[6:34], tree_set_name))
sys.stdout.flush()
try:
result = runBench(bench, *benchmark_setup)
except SkippedTest:
print("skipped")
except KeyboardInterrupt:
print("interrupted by user")
sys.exit(1)
except Exception:
exc_type, exc_value = sys.exc_info()[:2]
print("failed: %s: %s" % (exc_type.__name__, exc_value))
exc_type = exc_value = None
else:
print("%9.4f msec/pass, best of (%s)" % (
min(result), ' '.join("%9.4f" % t for t in result)))
if len(benchmark_suites) > 1:
print('') # empty line between different benchmarks
############################################################
# Main program
############################################################
def main(benchmark_class):
import_lxml = True
callgrind_zero = False
if len(sys.argv) > 1:
try:
sys.argv.remove('-i')
# run benchmark 'inplace'
sys.path.insert(0, 'src')
except ValueError:
pass
try:
sys.argv.remove('-nolxml')
# run without lxml
import_lxml = False
except ValueError:
pass
try:
sys.argv.remove('-z')
# reset callgrind after tree setup
callgrind_zero = True
except ValueError:
pass
initArgs(sys.argv)
_etrees = []
if import_lxml:
from lxml import etree
_etrees.append(etree)
print("Using lxml %s (with libxml2 %s)" % (
etree.__version__, '.'.join(map(str, etree.LIBXML_VERSION))))
try:
sys.argv.remove('-fel')
except ValueError:
pass
else:
# use fast element creation in lxml.etree
etree.set_element_class_lookup(
etree.ElementDefaultClassLookup())
if len(sys.argv) > 1:
if '-a' in sys.argv or '-c' in sys.argv:
# 'all' or 'C-implementations' ?
try:
sys.argv.remove('-c')
except ValueError:
pass
try:
import cElementTree as cET
_etrees.append(cET)
except ImportError:
try:
import xml.etree.cElementTree as cET
_etrees.append(cET)
except ImportError:
pass
try:
# 'all' ?
sys.argv.remove('-a')
except ValueError:
pass
else:
try:
from elementtree import ElementTree as ET
_etrees.append(ET)
except ImportError:
try:
from xml.etree import ElementTree as ET
_etrees.append(ET)
except ImportError:
pass
if not _etrees:
print("No library to test. Exiting.")
sys.exit(1)
print("Running benchmarks in Python %s" % (sys.version_info,))
print("Preparing test suites and trees ...")
selected = set( sys.argv[1:] )
benchmark_suites, benchmarks = \
buildSuites(benchmark_class, _etrees, selected)
print("Running benchmark on", ', '.join(b.lib_name
for b in benchmark_suites))
print('')
printSetupTimes(benchmark_suites)
if callgrind_zero:
cmd = open("callgrind.cmd", 'w')
cmd.write('+Instrumentation\n')
cmd.write('Zero\n')
cmd.close()
runBenchmarks(benchmark_suites, benchmarks)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/buildlibxml.py 0000644 0001751 0000200 00000050041 15001705132 014737 0 ustar 00runner docker import json
import os, re, sys, subprocess, platform
import tarfile
import time
from distutils import log
from contextlib import closing, contextmanager
from ftplib import FTP
import urllib.error
from urllib.parse import urljoin, unquote, urlparse
from urllib.request import urlretrieve, urlopen, Request
multi_make_options = []
try:
import multiprocessing
cpus = multiprocessing.cpu_count()
if cpus > 1:
if cpus > 5:
cpus = 5
multi_make_options = ['-j%d' % (cpus+1)]
except:
pass
# overridable to control script usage
sys_platform = sys.platform
# use pre-built libraries on Windows
def download_and_extract_windows_binaries(destdir):
url = "https://api.github.com/repos/lxml/libxml2-win-binaries/releases?per_page=5"
releases, _ = read_url(
url,
accept="application/vnd.github+json",
as_json=True,
github_api_token=os.environ.get("GITHUB_API_TOKEN"),
)
max_release = {'tag_name': ''}
for release in releases:
if max_release['tag_name'] < release.get('tag_name', ''):
max_release = release
url = "https://github.com/lxml/libxml2-win-binaries/releases/download/%s/" % max_release['tag_name']
filenames = [asset['name'] for asset in max_release.get('assets', ())]
# Check for native ARM64 build or the environment variable that is set by
# Visual Studio for cross-compilation (same variable as setuptools uses)
if platform.machine() == 'ARM64' or os.getenv('VSCMD_ARG_TGT_ARCH') == 'arm64':
arch = "win-arm64"
elif sys.maxsize > 2**32:
arch = "win64"
else:
arch = "win32"
if sys.version_info < (3, 5):
arch = 'vs2008.' + arch
arch_part = '.' + arch + '.'
filenames = [filename for filename in filenames if arch_part in filename]
libs = {}
for libname in ['libxml2', 'libxslt', 'zlib', 'iconv']:
libs[libname] = "%s-%s.%s.zip" % (
libname,
find_max_version(libname, filenames),
arch,
)
if not os.path.exists(destdir):
os.makedirs(destdir)
for libname, libfn in libs.items():
srcfile = urljoin(url, libfn)
destfile = os.path.join(destdir, libfn)
if os.path.exists(destfile + ".keep"):
print('Using local copy of "{}"'.format(srcfile))
else:
print('Retrieving "%s" to "%s"' % (srcfile, destfile))
urlretrieve(srcfile, destfile)
d = unpack_zipfile(destfile, destdir)
libs[libname] = d
return libs
def find_top_dir_of_zipfile(zipfile):
topdir = None
files = [f.filename for f in zipfile.filelist]
dirs = [d for d in files if d.endswith('/')]
if dirs:
dirs.sort(key=len)
topdir = dirs[0]
topdir = topdir[:topdir.index("/")+1]
for path in files:
if not path.startswith(topdir):
topdir = None
break
assert topdir, (
"cannot determine single top-level directory in zip file %s" %
zipfile.filename)
return topdir.rstrip('/')
def unpack_zipfile(zipfn, destdir):
assert zipfn.endswith('.zip')
import zipfile
print('Unpacking %s into %s' % (os.path.basename(zipfn), destdir))
f = zipfile.ZipFile(zipfn)
try:
extracted_dir = os.path.join(destdir, find_top_dir_of_zipfile(f))
f.extractall(path=destdir)
finally:
f.close()
assert os.path.exists(extracted_dir), 'missing: %s' % extracted_dir
return extracted_dir
def get_prebuilt_libxml2xslt(download_dir, static_include_dirs, static_library_dirs):
assert sys_platform.startswith('win')
libs = download_and_extract_windows_binaries(download_dir)
for libname, path in libs.items():
i = os.path.join(path, 'include')
l = os.path.join(path, 'lib')
assert os.path.exists(i), 'does not exist: %s' % i
assert os.path.exists(l), 'does not exist: %s' % l
static_include_dirs.append(i)
static_library_dirs.append(l)
## Routines to download and build libxml2/xslt from sources:
LIBXML2_LOCATION = 'https://download.gnome.org/sources/libxml2/'
LIBXSLT_LOCATION = 'https://download.gnome.org/sources/libxslt/'
LIBICONV_LOCATION = 'https://ftp.gnu.org/pub/gnu/libiconv/'
ZLIB_LOCATION = 'https://zlib.net/'
match_libfile_version = re.compile('^[^-]*-([.0-9-]+)[.].*').match
def _find_content_encoding(response, default='iso8859-1'):
from email.message import Message
content_type = response.headers.get('Content-Type')
if content_type:
msg = Message()
msg.add_header('Content-Type', content_type)
charset = msg.get_content_charset(default)
else:
charset = default
return charset
def remote_listdir(url):
try:
return _list_dir_urllib(url)
except IOError:
assert url.lower().startswith('ftp://')
print("Requesting with urllib failed. Falling back to ftplib. "
"Proxy argument will be ignored for %s" % url)
return _list_dir_ftplib(url)
def _list_dir_ftplib(url):
parts = urlparse(url)
ftp = FTP(parts.netloc)
try:
ftp.login()
ftp.cwd(parts.path)
data = []
ftp.dir(data.append)
finally:
ftp.quit()
return parse_text_ftplist("\n".join(data))
def read_url(url, decode=True, accept=None, as_json=False, github_api_token=None):
headers = {'User-Agent': 'https://github.com/lxml/lxml'}
if accept:
headers['Accept'] = accept
if github_api_token:
headers['authorization'] = "Bearer " + github_api_token
request = Request(url, headers=headers)
with closing(urlopen(request)) as res:
charset = _find_content_encoding(res)
content_type = res.headers.get('Content-Type')
data = res.read()
if decode:
data = data.decode(charset)
if as_json:
data = json.loads(data)
return data, content_type
def _list_dir_urllib(url):
data, content_type = read_url(url)
if content_type and content_type.startswith('text/html'):
files = parse_html_filelist(data)
else:
files = parse_text_ftplist(data)
return files
def http_find_latest_version_directory(url, version=None):
data, _ = read_url(url)
# e.g.
directories = [
(int(v[0]), int(v[1]))
for v in re.findall(r' href=["\']([0-9]+)\.([0-9]+)/?["\']', data)
]
if not directories:
return url
best_version = max(directories)
if version:
major, minor, _ = version.split(".", 2)
major, minor = int(major), int(minor)
if (major, minor) in directories:
best_version = (major, minor)
latest_dir = "%s.%s" % best_version
return urljoin(url, latest_dir) + "/"
def http_listfiles(url, re_pattern):
data, _ = read_url(url)
files = re.findall(re_pattern, data)
return files
def parse_text_ftplist(s):
for line in s.splitlines():
if not line.startswith('d'):
# -rw-r--r-- 1 ftp ftp 476 Sep 1 2011 md5sum.txt
# Last (9th) element is 'md5sum.txt' in the above example, but there
# may be variations, so we discard only the first 8 entries.
yield line.split(None, 8)[-1]
def parse_html_filelist(s):
re_href = re.compile(
r''']*\shref=["']([^;?"']+?)[;?"']''',
re.I|re.M)
links = set(re_href.findall(s))
for link in links:
if not link.endswith('/'):
yield unquote(link)
def tryint(s):
try:
return int(s)
except ValueError:
return s
@contextmanager
def py2_tarxz(filename):
import tempfile
with tempfile.TemporaryFile() as tmp:
subprocess.check_call(["xz", "-dc", filename], stdout=tmp.fileno())
tmp.seek(0)
with closing(tarfile.TarFile(fileobj=tmp)) as tf:
yield tf
def download_libxml2(dest_dir, version=None):
"""Downloads libxml2, returning the filename where the library was downloaded"""
#version_re = re.compile(r'LATEST_LIBXML2_IS_([0-9.]+[0-9](?:-[abrc0-9]+)?)')
version_re = re.compile(r'libxml2-([0-9.]+[0-9])[.]tar[.]xz')
filename = 'libxml2-%s.tar.xz'
if version == "2.9.12":
# Temporarily using the latest master (2.9.12+) until there is a release that supports lxml again.
from_location = "https://gitlab.gnome.org/GNOME/libxml2/-/archive/dea91c97debeac7c1aaf9c19f79029809e23a353/"
version = "dea91c97debeac7c1aaf9c19f79029809e23a353"
else:
from_location = http_find_latest_version_directory(LIBXML2_LOCATION, version=version)
return download_library(dest_dir, from_location, 'libxml2',
version_re, filename, version=version)
def download_libxslt(dest_dir, version=None):
"""Downloads libxslt, returning the filename where the library was downloaded"""
#version_re = re.compile(r'LATEST_LIBXSLT_IS_([0-9.]+[0-9](?:-[abrc0-9]+)?)')
version_re = re.compile(r'libxslt-([0-9.]+[0-9])[.]tar[.]xz')
filename = 'libxslt-%s.tar.xz'
from_location = http_find_latest_version_directory(LIBXSLT_LOCATION, version=version)
return download_library(dest_dir, from_location, 'libxslt',
version_re, filename, version=version)
def download_libiconv(dest_dir, version=None):
"""Downloads libiconv, returning the filename where the library was downloaded"""
version_re = re.compile(r'libiconv-([0-9.]+[0-9])[.]tar[.]gz')
filename = 'libiconv-%s.tar.gz'
return download_library(dest_dir, LIBICONV_LOCATION, 'libiconv',
version_re, filename, version=version)
def download_zlib(dest_dir, version):
"""Downloads zlib, returning the filename where the library was downloaded"""
version_re = re.compile(r'zlib-([0-9.]+[0-9])[.]tar[.]gz')
filename = 'zlib-%s.tar.gz'
return download_library(dest_dir, ZLIB_LOCATION, 'zlib',
version_re, filename, version=version)
def find_max_version(libname, filenames, version_re=None):
if version_re is None:
version_re = re.compile(r'%s-([0-9.]+[0-9](?:-[abrc0-9]+)?)' % libname)
versions = []
for fn in filenames:
match = version_re.search(fn)
if match:
version_string = match.group(1)
versions.append((tuple(map(tryint, version_string.replace("-", ".-").split('.'))),
version_string))
if not versions:
raise Exception(
"Could not find the most current version of %s from the files: %s" % (
libname, filenames))
versions.sort()
version_string = versions[-1][-1]
print('Latest version of %s is %s' % (libname, version_string))
return version_string
def download_library(dest_dir, location, name, version_re, filename, version=None):
if version is None:
try:
if location.startswith('ftp://'):
fns = list(remote_listdir(location))
else:
fns = http_listfiles(location, '(%s)' % filename.replace('%s', '(?:[0-9.]+[0-9])'))
print(f"Found {len(fns)} links at {location}")
version = find_max_version(name, fns, version_re)
except IOError:
# network failure - maybe we have the files already?
latest = (0,0,0)
fns = os.listdir(dest_dir)
for fn in fns:
if fn.startswith(name+'-'):
match = match_libfile_version(fn)
if match:
version_tuple = tuple(map(tryint, match.group(1).split('.')))
if version_tuple > latest:
latest = version_tuple
filename = fn
version = None
if latest == (0,0,0):
raise
if version:
filename = filename % version
full_url = urljoin(location, filename)
dest_filename = os.path.join(dest_dir, filename)
if os.path.exists(dest_filename):
print(('Using existing %s downloaded into %s '
'(delete this file if you want to re-download the package)') % (
name, dest_filename))
return dest_filename
print('Downloading %s into %s from %s' % (name, dest_filename, full_url))
try:
urlretrieve(full_url, dest_filename)
except urllib.error.URLError as exc:
# retry once
retry_after_seconds = 2
print(f"Download failed: {exc}, retrying in {int(retry_after_seconds)} seconds…")
time.sleep(retry_after_seconds)
urlretrieve(full_url, dest_filename)
return dest_filename
def unpack_tarball(tar_filename, dest):
print('Unpacking %s into %s' % (os.path.basename(tar_filename), dest))
if sys.version_info[0] < 3 and tar_filename.endswith('.xz'):
# Py 2.7 lacks lzma support
tar_cm = py2_tarxz(tar_filename)
else:
tar_cm = closing(tarfile.open(tar_filename))
base_dir = None
with tar_cm as tar:
for member in tar:
base_name = member.name.split('/')[0]
if base_dir is None:
base_dir = base_name
elif base_dir != base_name:
print('Unexpected path in %s: %s' % (tar_filename, base_name))
tar.extractall(dest)
return os.path.join(dest, base_dir)
def call_subprocess(cmd, **kw):
import subprocess
cwd = kw.get('cwd', '.')
cmd_desc = ' '.join(cmd)
log.info('Running "%s" in %s' % (cmd_desc, cwd))
returncode = subprocess.call(cmd, **kw)
if returncode:
raise Exception('Command "%s" returned code %s' % (cmd_desc, returncode))
def safe_mkdir(dir):
if not os.path.exists(dir):
os.makedirs(dir)
def cmmi(configure_cmd, build_dir, multicore=None, **call_setup):
print('Starting build in %s' % build_dir)
call_subprocess(configure_cmd, cwd=build_dir, **call_setup)
if not multicore:
make_jobs = multi_make_options
elif int(multicore) > 1:
make_jobs = ['-j%s' % multicore]
else:
make_jobs = []
call_subprocess(
['make'] + make_jobs,
cwd=build_dir, **call_setup)
call_subprocess(
['make'] + make_jobs + ['install'],
cwd=build_dir, **call_setup)
def configure_darwin_env(env_setup):
import platform
# configure target architectures on MacOS-X (x86_64 + Arm64, by default)
major_version, minor_version = tuple(map(int, platform.mac_ver()[0].split('.')[:2]))
if major_version >= 11:
env_default = {
'CFLAGS': "-arch x86_64 -arch arm64 -O3",
'LDFLAGS': "-arch x86_64 -arch arm64",
'MACOSX_DEPLOYMENT_TARGET': "11.0"
}
env_default.update(os.environ)
env_setup['env'] = env_default
def build_libxml2xslt(download_dir, build_dir,
static_include_dirs, static_library_dirs,
static_cflags, static_binaries,
libxml2_version=None,
libxslt_version=None,
libiconv_version=None,
zlib_version=None,
multicore=None):
safe_mkdir(download_dir)
safe_mkdir(build_dir)
zlib_dir = unpack_tarball(download_zlib(download_dir, zlib_version), build_dir)
libiconv_dir = unpack_tarball(download_libiconv(download_dir, libiconv_version), build_dir)
libxml2_dir = unpack_tarball(download_libxml2(download_dir, libxml2_version), build_dir)
libxslt_dir = unpack_tarball(download_libxslt(download_dir, libxslt_version), build_dir)
prefix = os.path.join(os.path.abspath(build_dir), 'libxml2')
lib_dir = os.path.join(prefix, 'lib')
safe_mkdir(prefix)
lib_names = ['libxml2', 'libexslt', 'libxslt', 'iconv', 'libz']
existing_libs = {
lib: os.path.join(lib_dir, filename)
for lib in lib_names
for filename in os.listdir(lib_dir)
if lib in filename and filename.endswith('.a')
} if os.path.isdir(lib_dir) else {}
def has_current_lib(name, build_dir, _build_all_following=[False]):
if _build_all_following[0]:
return False # a dependency was rebuilt => rebuilt this lib as well
lib_file = existing_libs.get(name)
found = lib_file and os.path.getmtime(lib_file) > os.path.getmtime(build_dir)
if found:
print("Found pre-built '%s'" % name)
else:
# also rebuild all following libs (which may depend on this one)
_build_all_following[0] = True
return found
call_setup = {}
if sys_platform == 'darwin':
configure_darwin_env(call_setup)
configure_cmd = ['./configure',
'--disable-dependency-tracking',
'--disable-shared',
'--prefix=%s' % prefix,
]
# build zlib
zlib_configure_cmd = [
'./configure',
'--prefix=%s' % prefix,
]
if not has_current_lib("libz", zlib_dir):
cmmi(zlib_configure_cmd, zlib_dir, multicore, **call_setup)
# build libiconv
if not has_current_lib("iconv", libiconv_dir):
cmmi(configure_cmd, libiconv_dir, multicore, **call_setup)
# build libxml2
libxml2_configure_cmd = configure_cmd + [
'--without-python',
'--with-http',
'--with-ftp',
'--with-iconv=%s' % prefix,
'--with-zlib=%s' % prefix,
]
if not libxml2_version:
libxml2_version = os.path.basename(libxml2_dir).split('-', 1)[-1]
if tuple(map(tryint, libxml2_version.split('-', 1)[0].split('.'))) >= (2, 9, 5):
libxml2_configure_cmd.append('--without-lzma') # can't currently build that
try:
if tuple(map(tryint, libxml2_version.split('-', 1)[0].split('.'))) >= (2, 7, 3):
libxml2_configure_cmd.append('--enable-rebuild-docs=no')
except Exception:
pass # this isn't required, so ignore any errors
if not has_current_lib("libxml2", libxml2_dir):
if not os.path.exists(os.path.join(libxml2_dir, "configure")):
# Allow building from git sources by running autoconf etc.
libxml2_configure_cmd[0] = "./autogen.sh"
cmmi(libxml2_configure_cmd, libxml2_dir, multicore, **call_setup)
# Fix up libxslt configure script (needed up to and including 1.1.34)
# https://gitlab.gnome.org/GNOME/libxslt/-/commit/90c34c8bb90e095a8a8fe8b2ce368bd9ff1837cc
with open(os.path.join(libxslt_dir, "configure"), 'rb') as f:
config_script = f.read()
if b' --libs print ' in config_script:
config_script = config_script.replace(b' --libs print ', b' --libs ')
with open(os.path.join(libxslt_dir, "configure"), 'wb') as f:
f.write(config_script)
# build libxslt
libxslt_configure_cmd = configure_cmd + [
'--without-python',
'--with-libxml-prefix=%s' % prefix,
'--without-crypto',
]
if not (has_current_lib("libxslt", libxslt_dir) and has_current_lib("libexslt", libxslt_dir)):
cmmi(libxslt_configure_cmd, libxslt_dir, multicore, **call_setup)
# collect build setup for lxml
xslt_config = os.path.join(prefix, 'bin', 'xslt-config')
xml2_config = os.path.join(prefix, 'bin', 'xml2-config')
static_include_dirs.extend([
os.path.join(prefix, 'include'),
os.path.join(prefix, 'include', 'libxml2'),
os.path.join(prefix, 'include', 'libxslt'),
os.path.join(prefix, 'include', 'libexslt')])
static_library_dirs.append(lib_dir)
listdir = os.listdir(lib_dir)
static_binaries += [os.path.join(lib_dir, filename)
for lib in lib_names
for filename in listdir
if lib in filename and filename.endswith('.a')]
return xml2_config, xslt_config
def main():
static_include_dirs = []
static_library_dirs = []
download_dir = "libs"
if sys_platform.startswith('win'):
return get_prebuilt_libxml2xslt(
download_dir, static_include_dirs, static_library_dirs)
else:
return build_libxml2xslt(
download_dir, 'build/tmp',
static_include_dirs, static_library_dirs,
static_cflags=[],
static_binaries=[]
)
if __name__ == '__main__':
if len(sys.argv) > 1:
# change global sys_platform setting
sys_platform = sys.argv[1]
main()
././@PaxHeader 0000000 0000000 0000000 00000000031 00000000000 010207 x ustar 00 25 mtime=1745324802.6226
lxml-5.4.0/doc/ 0000755 0001751 0000200 00000000000 15001705403 012624 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/FAQ.txt 0000644 0001751 0000200 00000161171 15001705132 014002 0 ustar 00runner docker =====================================
lxml FAQ - Frequently Asked Questions
=====================================
.. meta::
:description: Frequently Asked Questions about lxml (FAQ)
:keywords: lxml, lxml.etree, FAQ, frequently asked questions
Frequently asked questions on lxml. See also the notes on compatibility_ to
ElementTree_.
.. _compatibility: compatibility.html
.. _ElementTree: http://effbot.org/zone/element-index.htm
.. _`build instructions`: build.html
.. _`MacOS-X` : build.html#building-lxml-on-macos-x
.. contents::
..
1 General Questions
1.1 Is there a tutorial?
1.2 Where can I find more documentation about lxml?
1.3 What standards does lxml implement?
1.4 Who uses lxml?
1.5 What is the difference between lxml.etree and lxml.objectify?
1.6 How can I make my application run faster?
1.7 What about that trailing text on serialised Elements?
1.8 How can I find out if an Element is a comment or PI?
1.9 How can I map an XML tree into a dict of dicts?
1.10 Why does lxml sometimes return 'str' values for text in Python 2?
1.11 Why do I get XInclude or DTD lookup failures on some systems but not on others?
1.12 How do namespaces work in lxml?
2 Installation
2.1 Which version of libxml2 and libxslt should I use or require?
2.2 Where are the binary builds?
2.3 Why do I get errors about missing UCS4 symbols when installing lxml?
2.4 My C compiler crashes on installation
3 Contributing
3.1 Why is lxml not written in Python?
3.2 How can I contribute?
4 Bugs
4.1 My application crashes!
4.2 My application crashes on MacOS-X!
4.3 I think I have found a bug in lxml. What should I do?
4.4 How do I know a bug is really in lxml and not in libxml2?
4.5 My application crashes with xmlsec!
5 Threading
5.1 Can I use threads to concurrently access the lxml API?
5.2 Does my program run faster if I use threads?
5.3 Would my single-threaded program run faster if I turned off threading?
5.4 Why can't I reuse XSLT stylesheets in other threads?
5.5 My program crashes when run with mod_python/Pyro/Zope/Plone/...
6 Parsing and Serialisation
6.1 Why doesn't the ``pretty_print`` option reformat my XML output?
6.2 Why can't lxml parse my XML from unicode strings?
6.3 Can lxml parse from file objects opened in unicode mode?
6.4 What is the difference between str(xslt(doc)) and xslt(doc).write() ?
6.5 Why can't I just delete parents or clear the root node in iterparse()?
6.6 How do I output null characters in XML text?
6.7 Is lxml vulnerable to XML bombs?
6.8 How do I configure lxml safely as a web-service endpoint?
6.9 How can I sort the attributes?
7 XPath and Document Traversal
7.1 What are the ``findall()`` and ``xpath()`` methods on Element(Tree)?
7.2 Why doesn't ``findall()`` support full XPath expressions?
7.3 How can I find out which namespace prefixes are used in a document?
7.4 How can I specify a default namespace for XPath expressions?
7.5 How can I modify the tree during iteration?
The code examples below use the `'lxml.etree`` module:
.. sourcecode:: pycon
>>> from lxml import etree
..
>>> import sys
>>> _etree = etree
>>> if sys.version_info[0] >= 3:
... class etree_mock(object):
... def __getattr__(self, name): return getattr(_etree, name)
... def tostring(self, *args, **kwargs):
... s = _etree.tostring(*args, **kwargs)
... if isinstance(s, bytes): s = s.decode("utf-8") # CR
... if s[-1] == '\n': s = s[:-1]
... return s
... else:
... class etree_mock(object):
... def __getattr__(self, name): return getattr(_etree, name)
... def tostring(self, *args, **kwargs):
... s = _etree.tostring(*args, **kwargs)
... if s[-1] == '\n': s = s[:-1]
... return s
>>> etree = etree_mock()
General Questions
=================
Is there a tutorial?
--------------------
Read the `lxml.etree Tutorial`_. While this is still work in progress
(just as any good documentation), it provides an overview of the most
important concepts in ``lxml.etree``. If you want to help out,
improving the tutorial is a very good place to start.
There is also a `tutorial for ElementTree`_ which works for
``lxml.etree``. The documentation of the `extended etree API`_ also
contains many examples for ``lxml.etree``. Fredrik Lundh's `element
library`_ contains a lot of nice recipes that show how to solve common
tasks in ElementTree and lxml.etree. To learn using
``lxml.objectify``, read the `objectify documentation`_.
John Shipman has written another tutorial called `Python XML
processing with lxml`_ that contains lots of examples. Liza Daly
wrote a nice article about high-performance aspects when `parsing
large files with lxml`_.
.. _`lxml.etree Tutorial`: tutorial.html
.. _`tutorial for ElementTree`: https://web.archive.org/web/20200720191942/https://effbot.org/zone/element.htm
.. _`extended etree API`: api.html
.. _`objectify documentation`: objectify.html
.. _`Python XML processing with lxml`: https://web.archive.org/web/20190522191656/http://infohost.nmt.edu/tcc/help/pubs/pylxml/web/index.html
.. _`element library`: https://web.archive.org/web/20200703234431/http://www.effbot.org/zone/element-lib.htm
.. _`parsing large files with lxml`: http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
Where can I find more documentation about lxml?
-----------------------------------------------
There is a lot of documentation on the web and also in the Python
standard library documentation, as lxml implements the well-known
`ElementTree API`_ and tries to follow its documentation as closely as
possible. The recipes in Fredrik Lundh's `element library`_ are
generally worth taking a look at. There are a couple of issues where
lxml cannot keep up compatibility. They are described in the
compatibility_ documentation.
The lxml specific extensions to the API are described by individual
files in the ``doc`` directory of the source distribution and on `the
web page`_.
The `generated API documentation`_ is a comprehensive API reference
for the lxml package.
.. _`ElementTree API`: https://web.archive.org/web/20200703191710/http://www.effbot.org/zone/element-index.htm
.. _`the web page`: https://lxml.de/#documentation
.. _`generated API documentation`: api/index.html
What standards does lxml implement?
-----------------------------------
The compliance to XML Standards depends on the support in libxml2 and libxslt.
Here is a quote from `http://xmlsoft.org/ `_
support, there is a tool called `rnc2rng `_,
a CSS parser and toolkit, can be used with ``lxml.cssselect``
* `Deliverance `_,
a content theming tool
* `Enfold Proxy 4 `_,
a web server accelerator with on-the-fly XSLT processing
* `Inteproxy `_,
a secure HTTP proxy
* `lwebstring `_,
an XML template engine
* `openpyxl `_,
a library for handling OpenXML document meta data
* `PsychoPy `_,
a WSGI web development framework based on XML pipelines
* `pycsw `_,
an `OGC CSW `_ server implementation written in Python
* `PyQuery `_,
a query framework for XML/HTML, similar to jQuery for JavaScript
* `python-docx `_,
a package for handling Microsoft's Word OpenXML format
* `Rambler `_,
an RDFa parser with a simple dictionary-like interface.
* `xupdate-processor `_,
an XUpdate implementation for lxml.etree
* `Diazo `_,
Zope3 interface bindings for lxml
* `z3c.rml `_,
an implementation of ReportLab's RML format
* `zif.sedna `_,
an XQuery based interface to the Sedna OpenSource XML database
And don't miss the quotes by our generally happy_ users_, and other
`sites that link to lxml`_. As `Liza Daly`_ puts it: "Many software
products come with the pick-two caveat, meaning that you must choose
only two: speed, flexibility, or readability. When used carefully,
lxml can provide all three."
.. _Zope: http://www.zope.org/
.. _Plone: http://www.plone.org/
.. _Django: https://www.djangoproject.com/
.. _happy: http://thread.gmane.org/gmane.comp.python.lxml.devel/3244/focus=3244
.. _users: http://article.gmane.org/gmane.comp.python.lxml.devel/3246
.. _`sites that link to lxml`: http://www.google.com/search?as_lq=http:%2F%2Flxml.de%2F
.. _`Liza Daly`: http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
What is the difference between lxml.etree and lxml.objectify?
-------------------------------------------------------------
The two modules provide different ways of handling XML. However, objectify
builds on top of lxml.etree and therefore inherits most of its capabilities
and a large portion of its API.
* lxml.etree is a generic API for XML and HTML handling. It aims for
ElementTree compatibility_ and supports the entire XML infoset. It is well
suited for both mixed content and data centric XML. Its generality makes it
the best choice for most applications.
* lxml.objectify is a specialized API for XML data handling in a Python object
syntax. It provides a very natural way to deal with data fields stored in a
structurally well defined XML format. Data is automatically converted to
Python data types and can be manipulated with normal Python operators. Look
at the examples in the `objectify documentation`_ to see what it feels like
to use it.
Objectify is not well suited for mixed contents or HTML documents. As it is
built on top of lxml.etree, however, it inherits the normal support for
XPath, XSLT or validation.
How can I make my application run faster?
-----------------------------------------
lxml.etree is a very fast library for processing XML. There are, however, `a
few caveats`_ involved in the mapping of the powerful libxml2 library to the
simple and convenient ElementTree API. Not all operations are as fast as the
simplicity of the API might suggest, while some use cases can heavily benefit
from finding the right way of doing them. The `benchmark page`_ has a
comparison to other ElementTree implementations and a number of tips for
performance tweaking. As with any Python application, the rule of thumb is:
the more of your processing runs in C, the faster your application gets. See
also the section on threading_.
.. _`a few caveats`: performance.html#the-elementtree-api
.. _`benchmark page`: performance.html
.. _threading: #threading
What about that trailing text on serialised Elements?
-----------------------------------------------------
The ElementTree tree model defines an Element as a container with a tag name,
contained text, child Elements and a tail text. This means that whenever you
serialise an Element, you will get all parts of that Element:
.. sourcecode:: pycon
>>> root = etree.XML("text tailtext tail
Here is an example that shows why not serialising the tail would be
even more surprising from an object point of view:
.. sourcecode:: pycon
>>> root = etree.Element("test")
>>> root.text = "TEXT"
>>> print(etree.tostring(root))
TEXT
>>> root.tail = "TAIL"
>>> print(etree.tostring(root))
TEXT TAIL
>>> root.tail = None
>>> print(etree.tostring(root))
TEXT
Just imagine a Python list where you append an item and it doesn't
show up when you look at the list.
The ``.tail`` property is a huge simplification for the tree model as
it avoids text nodes to appear in the list of children and makes
access to them quick and simple. So this is a benefit in most
applications and simplifies many, many XML tree algorithms.
However, in document-like XML (and especially HTML), the above result can be
unexpected to new users and can sometimes require a bit more overhead. A good
way to deal with this is to use helper functions that copy the Element without
its tail. The ``lxml.html`` package also deals with this in a couple of
places, as most HTML algorithms benefit from a tail-free behaviour.
How can I find out if an Element is a comment or PI?
----------------------------------------------------
.. sourcecode:: pycon
>>> root = etree.XML("`_.
Many systems have them installed by default, but some don't.
On Linux systems, the default place to look is the index file
``/etc/xml/catalog``, which most importantly provides a mapping from
doctype IDs to locally installed DTD files.
See the `libxml2 catalogue documentation `_
for further information.
How do namespaces work in lxml?
-------------------------------
The same as in ElementTree. See the `tutorial `_.
Installation
============
Which version of libxml2 and libxslt should I use or require?
-------------------------------------------------------------
It really depends on your application, but the rule of thumb is: more recent
versions contain less bugs and provide more features.
* Do not use the stock libxml2 versions 2.9.11 or 2.9.12. They are incompatible
with lxml and lead to excess output on serialisation. For static builds
against 2.9.12, lxml automatically downloads a post-release version that
contains a work-around.
* Try to use versions of both libraries that were released together. At least
the libxml2 version should not be older than the libxslt version.
* If you use XML Schema or Schematron which are still under development, the
most recent version of libxml2 is usually a good bet.
* The same applies to XPath, where a substantial number of bugs and memory
leaks were fixed over time. If you encounter crashes or memory leaks in
XPath applications, try a more recent version of libxml2.
* For the normal tree handling, however, any libxml2 version starting with
2.7.x should do.
Read the `release notes of libxml2`_ and the `release notes of libxslt`_ to
see when (or if) a specific bug has been fixed.
.. _`release notes of libxml2`: http://xmlsoft.org/news.html
.. _`release notes of libxslt`: http://xmlsoft.org/XSLT/news.html
Where are the binary builds?
----------------------------
We provide binaries for Linux (`manylinux`_), macOS and MS Windows
shortly after each source release.
Thanks to the help by Maximilian Hils and the Appveyor build service,
we try to serve the frequent requests for binary builds available
for Microsoft Windows in a timely fashion, since users of that platform
usually fail to build lxml themselves. Two of the major design issues
of this operating system make this non-trivial for its users: the lack
of a pre-installed standard C-compiler and the missing package management.
We currently rely on the `WinLibs project `_
to provide library versions that are buildable on MS Windows. If the library
that we use in lxml's Windows binary wheels is outdated, it is probably because
they have not updated their repositories yet. Consider filing a ticket on their
side and notifying us when a new version is available, so that we can integrate it.
.. _manylinux: https://www.python.org/dev/peps/pep-0513
Why do I get errors about missing UCS4 symbols when installing lxml?
--------------------------------------------------------------------
You are using a Python installation that was configured for a different
internal Unicode representation than the lxml package you are trying to
install. CPython versions before 3.3 allowed to switch between two types
at build time: the 32 bit encoding UCS4 and the 16 bit encoding UCS2.
Sadly, both are not compatible, so eggs and other binary distributions
can only support the one they were compiled with.
This means that you have to compile lxml from sources for your system. Note
that you do not need Cython for this, the lxml source distribution is directly
compilable on both platform types. See the `build instructions`_ on how to do
this.
My C compiler crashes on installation
-------------------------------------
lxml consists of a relatively large amount of (Cython) generated C code
in a single source module. Compiling this module requires a lot of free
memory, usually more than half a GB, which can pose problems especially on
shared/cloud build systems.
If your C compiler crashes while building lxml from sources, consider
using one of the binary wheels that we provide. The "manylinux_" binaries
should generally work well on most build systems and install substantially
faster than a source build.
Contributing
============
Why is lxml not written in Python?
----------------------------------
It *almost* is.
lxml is not written in plain Python, because it interfaces with two C
libraries: libxml2 and libxslt. Accessing them at the C-level is
required for performance reasons.
However, to avoid writing plain C-code and caring too much about the
details of built-in types and reference counting, lxml is written in
Cython_, a superset of the Python language that translates to C-code.
Chances are that if you know Python, you can write `code that Cython
accepts`_. Again, the C-ish style used in the lxml code is just for
performance optimisations. If you want to contribute, don't bother
with the details, a Python implementation of your contribution is
better than none. And keep in mind that lxml's flexible API often
favours an implementation of features in pure Python, without
bothering with C-code at all. For example, the ``lxml.html`` package
is written entirely in Python.
Please contact the `mailing list`_ if you need any help.
.. _Cython: http://cython.org/
.. _`code that Cython accepts`: http://docs.cython.org/docs/tutorial.html
How can I contribute?
---------------------
If you find something that you would like lxml to do (or do better),
then please tell us about it on the `mailing list`_. Pull requests
on github are always appreciated, especially when accompanied by unit
tests and documentation (doctests would be great). See the ``tests``
subdirectories in the lxml source tree (below the ``src`` directory)
and the ReST_ `text files`_ in the ``doc`` directory.
We also have a `list of missing features`_ that we would like to
implement but didn't due to lack if time. If *you* find the time,
patches are very welcome.
.. _ReST: http://docutils.sourceforge.net/rst.html
.. _`text files`: https://github.com/lxml/lxml/tree/master/doc
.. _`list of missing features`: https://github.com/lxml/lxml/blob/master/IDEAS.txt
Besides enhancing the code, there are a lot of places where you can help the
project and its user base. You can
* spread the word and write about lxml. Many users (especially new Python
users) have not yet heard about lxml, although our user base is constantly
growing. If you write your own blog and feel like saying something about
lxml, go ahead and do so. If we think your contribution or criticism is
valuable to other users, we may even put a link or a quote on the project
page.
* provide code examples for the general usage of lxml or specific problems
solved with lxml. Readable code is a very good way of showing how a library
can be used and what great things you can do with it. Again, if we hear
about it, we can set a link on the project page.
* work on the documentation. The web page is generated from a set of ReST_
`text files`_. It is meant both as a representative project page for lxml
and as a site for documenting lxml's API and usage. If you have questions
or an idea how to make it more readable and accessible while you are reading
it, please send a comment to the `mailing list`_.
* enhance the web site. We put some work into making the web site
usable, understandable and also easy to find, but there's always
things that can be done better. You may notice that we are not
top-ranked when searching the web for "Python and XML", so maybe you
have an idea how to improve that.
* help with the tutorial. A tutorial is the most important starting point for
new users, so it is important for us to provide an easy to understand guide
into lxml. As all documentation, the tutorial is work in progress, so we
appreciate every helping hand.
* improve the docstrings. lxml uses docstrings to support Python's integrated
online ``help()`` function. However, sometimes these are not sufficient to
grasp the details of the function in question. If you find such a place,
you can try to write up a better description and send it to the `mailing
list`_.
Bugs
====
My application crashes!
-----------------------
One of the goals of lxml is "no segfaults", so if there is no clear
warning in the documentation that you were doing something potentially
harmful, you have found a bug and we would like to hear about it.
Please report this bug to the `mailing list`_. See the section on bug
reporting to learn how to do that.
If your application (or e.g. your web container) uses threads, please
see the FAQ section on threading_ to check if you touch on one of the
potential pitfalls.
In any case, try to reproduce the problem with the latest versions of
libxml2 and libxslt. From time to time, bugs and race conditions are found
in these libraries, so a more recent version might already contain a fix for
your problem.
Remember: even if you see lxml appear in a crash stack trace, it is
not necessarily lxml that *caused* the crash.
If you are using the *``xmlsec``* library together with lxml, you have to
make sure that both use the same version of libxml2. The binary wheels of
lxml statically include a (usually recent) version of libxml2, whereas
xmlsec often depends on the systemwide installed libraries. If you get
crashes or unexpected behaviour when using both, please make sure that both
get to use the same libxml2 version. Anaconda/condaforge/etc. based installations
will usually come with matching C libraries. If you use xmlsec with the system
libraries, please build lxml from sources against those as well, e.g. by installing
the development packages of libxml2 and libxslt and then installing lxml with
``python -m pip install --no-binary lxml lxml``
My application crashes on MacOS-X!
----------------------------------
This was a common problem up to lxml 2.1.x. Since lxml 2.2, the only
officially supported way to use it on this platform is through a
static build against freshly downloaded versions of libxml2 and
libxslt. See the build instructions for `MacOS-X`_.
I think I have found a bug in lxml. What should I do?
-----------------------------------------------------
First, you should look at the `current developer changelog`_ to see if this
is a known problem that has already been fixed in the master branch since the
release you are using.
.. _`current developer changelog`: https://github.com/lxml/lxml/blob/master/CHANGES.txt
Also, the 'crash' section above has a few good advices what to try to see if
the problem is really in lxml - and not in your setup. Believe it or not,
that happens more often than you might think, especially when old libraries
or even multiple library versions are installed.
You should always try to reproduce the problem with the latest
versions of libxml2 and libxslt - and make sure they are used.
``lxml.etree`` can tell you what it runs with:
.. sourcecode:: python
import sys
from lxml import etree
print("%-20s: %s" % ('Python', sys.version_info))
print("%-20s: %s" % ('lxml.etree', etree.LXML_VERSION))
print("%-20s: %s" % ('libxml used', etree.LIBXML_VERSION))
print("%-20s: %s" % ('libxml compiled', etree.LIBXML_COMPILED_VERSION))
print("%-20s: %s" % ('libxslt used', etree.LIBXSLT_VERSION))
print("%-20s: %s" % ('libxslt compiled', etree.LIBXSLT_COMPILED_VERSION))
If you can figure that the problem is not in lxml but in the
underlying libxml2 or libxslt, you can ask right on the respective
mailing lists, which may considerably reduce the time to find a fix or
work-around. See the next question for some hints on how to do that.
Otherwise, we would really like to hear about it. Please report it to
the `bug tracker`_ or to the `mailing list`_ so that we can fix it.
It is very helpful in this case if you can come up with a short code
snippet that demonstrates your problem. If others can reproduce and
see the problem, it is much easier for them to fix it - and maybe even
easier for you to describe it and get people convinced that it really
is a problem to fix.
It is important that you always report the version of lxml, libxml2
and libxslt that you get from the code snippet above. If we do not
know the library versions you are using, we will ask back, so it will
take longer for you to get a helpful answer.
Since as a user of lxml you are likely a programmer, you might find
`this article on bug reports`_ an interesting read.
.. _`bug tracker`: https://bugs.launchpad.net/lxml/
.. _`mailing list`: https://lxml.de/mailinglist/
.. _`this article on bug reports`: http://www.chiark.greenend.org.uk/~sgtatham/bugs.html
How do I know a bug is really in lxml and not in libxml2?
---------------------------------------------------------
A large part of lxml's functionality is implemented by libxml2 and
libxslt, so problems that you encounter may be in one or the other.
Knowing the right place to ask will reduce the time it takes to fix
the problem, or to find a work-around.
Both libxml2 and libxslt come with their own command line frontends,
namely ``xmllint`` and ``xsltproc``. If you encounter problems with
XSLT processing for specific stylesheets or with validation for
specific schemas, try to run the XSLT with ``xsltproc`` or the
validation with ``xmllint`` respectively to find out if it fails there
as well. If it does, please report directly to the mailing lists of
the respective project, namely:
* `libxml2 mailing list `_
* `libxslt mailing list `_
On the other hand, everything that seems to be related to Python code,
including custom resolvers, custom XPath functions, etc. is likely
outside of the scope of libxml2/libxslt. If you encounter problems
here or you are not sure where there the problem may come from, please
ask on the lxml mailing list first.
In any case, a good explanation of the problem including some simple
test code and some input data will help us (or the libxml2 developers)
see and understand the problem, which largely increases your chance of
getting help. See the question above for a few hints on what is
helpful here.
Threading
=========
Can I use threads to concurrently access the lxml API?
------------------------------------------------------
Short answer: yes, if you use lxml 2.2 and later.
Since version 1.1, lxml frees the GIL (Python's global interpreter
lock) internally when parsing from disk and memory, as long as you use
either the default parser (which is replicated for each thread) or
create a parser for each thread yourself. lxml also allows
concurrency during validation (RelaxNG and XMLSchema) and XSL
transformation. You can share RelaxNG, XMLSchema and XSLT objects
between threads.
While you can also share parsers between threads, this will serialize
the access to each of them, so it is better to ``.copy()`` parsers or
to just use the default parser if you do not need any special
configuration. The same applies to the XPath evaluators, which use an
internal lock to protect their prepared evaluation contexts. It is
therefore best to use separate evaluator instances in threads.
Warning: Before lxml 2.2, and especially before 2.1, there were
various issues when moving subtrees between different threads, or when
applying XSLT objects from one thread to trees parsed or modified in
another. If you need code to run with older versions, you should
generally avoid modifying trees in other threads than the one it was
generated in. Although this should work in many cases, there are
certain scenarios where the termination of a thread that parsed a tree
can crash the application if subtrees of this tree were moved to other
documents. You should be on the safe side when passing trees between
threads if you either
- do not modify these trees and do not move their elements to other
trees, or
- do not terminate threads while the trees they parsed are still in
use (e.g. by using a fixed size thread-pool or long-running threads
in processing chains)
Since lxml 2.2, even multi-thread pipelines are supported. However,
note that it is more efficient to do all tree work inside one thread,
than to let multiple threads work on a tree one after the other. This
is because trees inherit state from the thread that created them,
which must be maintained when the tree is modified inside another
thread.
Does my program run faster if I use threads?
--------------------------------------------
Depends. The best way to answer this is timing and profiling.
The global interpreter lock (GIL) in Python serializes access to the
interpreter, so if the majority of your processing is done in Python
code (walking trees, modifying elements, etc.), your gain will be
close to zero. The more of your XML processing moves into lxml,
however, the higher your gain. If your application is bound by XML
parsing and serialisation, or by very selective XPath expressions and
complex XSLTs, your speedup on multi-processor machines can be
substantial.
See the question above to learn which operations free the GIL to support
multi-threading.
Would my single-threaded program run faster if I turned off threading?
----------------------------------------------------------------------
Possibly, yes. You can see for yourself by compiling lxml entirely
without threading support. Pass the ``--without-threading`` option to
setup.py when building lxml from source. You can also build libxml2
without pthread support (``--without-pthreads`` option), which may add
another bit of performance. Note that this will leave internal data
structures entirely without thread protection, so make sure you really
do not use lxml outside of the main application thread in this case.
Why can't I reuse XSLT stylesheets in other threads?
----------------------------------------------------
Since later lxml 2.0 versions, you can do this. There is some
overhead involved as the result document needs an additional cleanup
traversal when the input document and/or the stylesheet were created
in other threads. However, on a multi-processor machine, the gain of
freeing the GIL easily covers this drawback.
If you need even the last bit of performance, consider keeping (a copy
of) the stylesheet in thread-local storage, and try creating the input
document(s) in the same thread. And do not forget to benchmark your
code to see if the increased code complexity is really worth it.
My program crashes when run with mod_python/Pyro/Zope/Plone/...
---------------------------------------------------------------
These environments can use threads in a way that may not make it obvious when
threads are created and what happens in which thread. This makes it hard to
ensure lxml's threading support is used in a reliable way. Sadly, if problems
arise, they are as diverse as the applications, so it is difficult to provide
any generally applicable solution. Also, these environments are so complex
that problems become hard to debug and even harder to reproduce in a
predictable way. If you encounter crashes in one of these systems, but your
code runs perfectly when started by hand, the following gives you a few hints
for possible approaches to solve your specific problem:
* make sure you use recent versions of libxml2, libxslt and lxml. The
libxml2 developers keep fixing bugs in each release, and lxml also
tries to become more robust against possible pitfalls. So newer
versions might already fix your problem in a reliable way. Version
2.2 of lxml contains many improvements.
* make sure the library versions you installed are really used. Do
not rely on what your operating system tells you! Print the version
constants in ``lxml.etree`` from within your runtime environment to
make sure it is the case. This is especially a problem under
MacOS-X when newer library versions were installed in addition to
the outdated system libraries. Please read the bugs section
regarding MacOS-X in this FAQ.
* if you use ``mod_python``, try setting this option:
PythonInterpreter main_interpreter
There was a discussion on the mailing list about this problem:
http://comments.gmane.org/gmane.comp.python.lxml.devel/2942
* in a threaded environment, try to initially import ``lxml.etree``
from the main application thread instead of doing first-time imports
separately in each spawned worker thread. If you cannot control the
thread spawning of your web/application server, an import of
``lxml.etree`` in sitecustomize.py or usercustomize.py may still do
the trick.
* compile lxml without threading support by running ``setup.py`` with the
``--without-threading`` option. While this might be slower in certain
scenarios on multi-processor systems, it *might* also keep your application
from crashing, which should be worth more to you than peek performance.
Remember that lxml is fast anyway, so concurrency may not even be worth it.
* look out for fancy XSLT stuff like foreign document access or
passing in subtrees through XSLT variables. This might or might not
work, depending on your specific usage. Again, later versions of
lxml and libxslt provide safer support here.
* try copying trees at suspicious places in your code and working with
those instead of a tree shared between threads. Note that the
copying must happen inside the target thread to be effective, not in
the thread that created the tree. Serialising in one thread and
parsing in another is also a simple (and fast) way of separating
thread contexts.
* try keeping thread-local copies of XSLT stylesheets, i.e. one per thread,
instead of sharing one. Also see the question above.
* you can try to serialise suspicious parts of your code with explicit thread
locks, thus disabling the concurrency of the runtime system.
* report back on the mailing list to see if there are other ways to work
around your specific problems. Do not forget to report the version numbers
of lxml, libxml2 and libxslt you are using (see the question on reporting
a bug).
Note that most of these options will degrade performance and/or your
code quality. If you are unsure what to do, please ask on the mailing
list.
Parsing and Serialisation
=========================
..
making doctest happy:
>>> try: from StringIO import StringIO
... except ImportError: from io import StringIO # Py3
>>> filename = StringIO("some text
If you want to be sure all blank text is removed from an XML document
(or just more blank text than the parser does by itself), you have to
use either a DTD to tell the parser which whitespace it can safely
ignore, or remove the ignorable whitespace manually after parsing,
e.g. by setting all tail text to None:
.. sourcecode:: python
for element in root.iter():
element.tail = None
Fredrik Lundh also has a Python-level function for indenting XML by
appending whitespace to tags. It can be found on his `element library
recipes page `_.
Why can't lxml parse my XML from unicode strings?
-------------------------------------------------
First of all, XML is explicitly defined as a stream of bytes. It's not
Unicode text. Take a look at the `XML specification`_, it's all about byte
sequences and how to map them to text and structure. That leads to rule
number one: do not decode your XML data yourself. That's a part of the
work of an XML parser, and it does it very well. Just pass it your data as
a plain byte stream, it will always do the right thing, by specification.
This also includes not opening XML files in text mode. Make sure you always
use binary mode, or, even better, pass the file path into lxml's ``parse()``
function to let it do the file opening, reading and closing itself. This
is the most simple and most efficient way to do it.
That being said, lxml can read Python unicode strings and even tries to
support them if libxml2 does not. This is because there is one valid use
case for parsing XML from text strings: literal XML fragments in source
code.
However, if the unicode string declares an XML encoding internally
(````), parsing is bound to fail, as this encoding is
almost certainly not the real encoding used in Python unicode. The same is
true for HTML unicode strings that contain charset meta tags, although the
problems may be more subtle here. The libxml2 HTML parser may not be able
to parse the meta tags in broken HTML and may end up ignoring them, so even
if parsing succeeds, later handling may still fail with character encoding
errors. Therefore, parsing HTML from unicode strings is a much saner thing
to do than parsing XML from unicode strings.
Note that Python uses different encodings for unicode on different platforms,
so even specifying the real internal unicode encoding is not portable between
Python interpreters. Don't do it.
Python unicode strings with XML data that carry encoding information are
broken. lxml will not parse them. You must provide parsable data in a
valid encoding.
.. _`XML specification`: http://www.w3.org/TR/REC-xml/
Can lxml parse from file objects opened in unicode/text mode?
-------------------------------------------------------------
Technically, yes. However, you likely do not want to do that, because
it is extremely inefficient. The text encoding that libxml2 uses
internally is UTF-8, so parsing from a Unicode file means that Python
first reads a chunk of data from the file, then decodes it into a new
buffer, and then copies it into a new unicode string object, just to
let libxml2 make yet another copy while encoding it down into UTF-8
in order to parse it. It's clear that this involves a lot more
recoding and copying than when parsing straight from the bytes that
the file contains.
If you really know the encoding better than the parser (e.g. when
parsing HTML that lacks a content declaration), then instead of passing
an encoding parameter into the file object when opening it, create a
new instance of an XMLParser or HTMLParser and pass the encoding into
its constructor. Afterwards, use that parser for parsing, e.g. by
passing it into the ``etree.parse(file, parser)`` function. Remember
to open the file in binary mode (mode="rb"), or, if possible, prefer
passing the file path directly into ``parse()`` instead of an opened
Python file object.
What is the difference between str(xslt(doc)) and xslt(doc).write() ?
---------------------------------------------------------------------
The str() implementation of the XSLTResultTree class (a subclass of the
ElementTree class) knows about the output method chosen in the stylesheet
(xsl:output), write() doesn't. If you call write(), the result will be a
normal XML tree serialization in the requested encoding. Calling this method
may also fail for XSLT results that are not XML trees (e.g. string results).
If you call str(), it will return the serialized result as specified by the
XSL transform. This correctly serializes string results to encoded Python
strings and honours ``xsl:output`` options like ``indent``. This almost
certainly does what you want, so you should only use ``write()`` if you are
sure that the XSLT result is an XML tree and you want to override the encoding
and indentation options requested by the stylesheet.
Why can't I just delete parents or clear the root node in iterparse()?
----------------------------------------------------------------------
The ``iterparse()`` implementation is based on the libxml2 parser. It
requires the tree to be intact to finish parsing. If you delete or modify
parents of the current node, chances are you modify the structure in a way
that breaks the parser. Normally, this will result in a segfault. Please
refer to the `iterparse section`_ of the lxml API documentation to find out
what you can do and what you can't do.
.. _`iterparse section`: parsing.html#iterparse-and-iterwalk
How do I output null characters in XML text?
--------------------------------------------
Don't. What you would produce is not well-formed XML. XML parsers
will refuse to parse a document that contains null characters. The
right way to embed binary data in XML is using a text encoding such as
uuencode or base64.
Is lxml vulnerable to XML bombs?
--------------------------------
This has nothing to do with lxml itself, only with the parser of
libxml2. Since libxml2 version 2.7, the parser imposes hard security
limits on input documents to prevent DoS attacks with forged input
data. Since lxml 2.2.1, you can disable these limits with the
``huge_tree`` parser option if you need to parse *really* large,
trusted documents. All lxml versions will leave these restrictions
enabled by default.
Note that libxml2 versions of the 2.6 series do not restrict their
parser and are therefore vulnerable to DoS attacks.
Note also that these "hard limits" may still be high enough to
allow for excessive resource usage in a given use case. They are
compile time modifiable, so building your own library versions will
allow you to change the limits to your own needs. Also see the next
question.
How do I use lxml safely as a web-service endpoint?
---------------------------------------------------
XML based web-service endpoints are generally subject to several
types of attacks if they allow some kind of untrusted input.
From the point of view of the underlying XML tool, the most
obvious attacks try to send a relatively small amount of data
that induces a comparatively large resource consumption on the
receiver side.
First of all, make sure network access is not enabled for the XML
parser that you use for parsing untrusted content and that it is
not configured to load external DTDs. Otherwise, attackers can
try to trick the parser into an attempt to load external resources
that are overly slow or impossible to retrieve, thus wasting time
and other valuable resources on your server such as socket
connections. Note that you can register your own document loader
in lxml, which allows for fine-grained control over any read access
to resources.
Some of the most famous excessive content expansion attacks
use XML entity references. Luckily, entity expansion is mostly
useless for the data commonly sent through web services and
can simply be disabled, which rules out several types of
denial of service attacks at once. This also involves an attack
that reads local files from the server, as XML entities can be
defined to expand into the content of external resources.
Consequently, version 1.2 of the SOAP standard explicitly
disallows entity references in the XML stream.
To disable entity expansion, use an XML parser that is configured
with the option ``resolve_entities=False``. Then, after (or
while) parsing the document, use ``root.iter(etree.Entity)`` to
recursively search for entity references. If it contains any,
reject the entire input document with a suitable error response.
In lxml 3.x, you can also use the new DTD introspection API to
apply your own restrictions on input documents. Since version 5.x,
lxml disables the expansion of external entities (XXE) by default.
If you really want to allow loading external files into XML documents
using this functionality, you have to explicitly set
``resolve_entities=True``.
Another attack to consider is compression bombs. If you allow
compressed input into your web service, attackers can try to send
well forged highly repetitive and thus very well compressing input
that unpacks into a very large XML document in your server's main
memory, potentially a thousand times larger than the compressed
input data.
As a counter measure, either disable compressed input for your
web server, at least for untrusted sources, or use incremental
parsing with ``iterparse()`` instead of parsing the whole input
document into memory in one shot. That allows you to enforce
suitable limits on the input by applying semantic checks that
detect and prevent an illegitimate use of your service. If
possible, you can also use this to reduce the amount of data
that you need to keep in memory while parsing the document,
thus further reducing the possibility of an attacker to trick
your system into excessive resource usage.
Finally, please be aware that XPath suffers from the same
vulnerability as SQL when it comes to content injection. The
obvious fix is to not build any XPath expressions via string
formatting or concatenation when the parameters may come from
untrusted sources, and instead use XPath variables, which
safely expose their values to the evaluation engine.
The defusedxml_ package comes with an example setup and a wrapper
API for lxml that applies certain counter measures internally.
.. _defusedxml: https://github.com/tiran/defusedxml
How can I sort the attributes?
------------------------------
lxml preserves the order in which attributes were originally created.
There is one case in which this is difficult: when attributes are passed
in a dict or as keyword arguments to the `Element()` factory. Before Python
3.6, dicts had no predictable order.
Since Python 3.6, however, dicts also preserve the creation order of their keys,
and lxml makes use of that since release 4.4.
In earlier versions, lxml tries to assure at least reproducible output by
sorting the attributes from the dict before creating them. All sequential
ways to set attributes keep their order and do not apply sorting. Also,
OrderedDict instances are recognised and not sorted.
In cases where you cannot control the order in which attributes are created,
you can still change it before serialisation. To sort them by name, for example,
you can apply the following function:
.. sourcecode:: python
def sort_attributes(root):
for el in root.iter():
attrib = el.attrib
if len(attrib) > 1:
attributes = sorted(attrib.items())
attrib.clear()
attrib.update(attributes)
XPath and Document Traversal
============================
What are the ``findall()`` and ``xpath()`` methods on Element(Tree)?
--------------------------------------------------------------------
``findall()`` is part of the original `ElementTree API`_. It supports a
`simple subset of the XPath language`_, without predicates, conditions and
other advanced features. It is very handy for finding specific tags in a
tree. Another important difference is namespace handling, which uses the
``{namespace}tagname`` notation. This is not supported by XPath. The
findall, find and findtext methods are compatible with other ElementTree
implementations and allow writing portable code that runs on ElementTree,
cElementTree and lxml.etree.
``xpath()``, on the other hand, supports the complete power of the XPath
language, including predicates, XPath functions and Python extension
functions. The syntax is defined by the `XPath specification`_. If you need
the expressiveness and selectivity of XPath, the ``xpath()`` method, the
``XPath`` class and the ``XPathEvaluator`` are the best choice_.
.. _`simple subset of the XPath language`: http://effbot.org/zone/element-xpath.htm
.. _`XPath specification`: http://www.w3.org/TR/xpath
.. _choice: performance.html#xpath
Why doesn't ``findall()`` support full XPath expressions?
---------------------------------------------------------
It was decided that it is more important to keep compatibility with
ElementTree_ to simplify code migration between the libraries. The main
difference compared to XPath is the ``{namespace}tagname`` notation used in
``findall()``, which is not valid XPath.
ElementTree and lxml.etree use the same implementation, which assures 100%
compatibility. Note that ``findall()`` is `so fast`_ in lxml that a native
implementation would not bring any performance benefits.
.. _`so fast`: performance.html#tree-traversal
How can I find out which namespace prefixes are used in a document?
-------------------------------------------------------------------
You can traverse the document (``root.iter()``) and collect the prefix
attributes from all Elements into a set. However, it is unlikely that you
really want to do that. You do not need these prefixes, honestly. You only
need the namespace URIs. All namespace comparisons use these, so feel free to
make up your own prefixes when you use XPath expressions or extension
functions.
The only place where you might consider specifying prefixes is the
serialization of Elements that were created through the API. Here, you can
specify a prefix mapping through the ``nsmap`` argument when creating the root
Element. Its children will then inherit this prefix for serialization.
How can I specify a default namespace for XPath expressions?
------------------------------------------------------------
You can't. In XPath 1.0, there is no such thing as a default namespace. Just
use an arbitrary prefix and let the namespace dictionary of the XPath evaluators
map it to your namespace. See also the question above.
How can I modify the tree during iteration?
-------------------------------------------
lxml's iterators need to hold on to an element in the tree in order to remember
their current position. Therefore, tree modifications between two calls into the
iterator can lead to surprising results if such an element is deleted or moved
around, for example.
If your code risks modifying elements that the iterator might still need, and
you know that the number of elements returned by the iterator is small, then just
read them all into a list (or use ``.findall()``), and iterate over that list.
If the number of elements can be larger and you really want to process the tree
incrementally, you can often use a read-ahead generator to make the iterator
advance beyond the critical point before touching the tree structure.
For example:
.. sourcecode:: python
from itertools import islice
from collections import deque
def readahead(iterator, count=1):
iterator = iter(iterator) # allow iterables as well
elements = deque(islice(iterator, 0, count))
for element in iterator:
elements.append(element)
yield elements.popleft()
yield from elements
for element in readahead(root.iterfind("path/to/children")):
element.getparent().remove(element)
././@PaxHeader 0000000 0000000 0000000 00000000031 00000000000 010207 x ustar 00 25 mtime=1745324802.6226
lxml-5.4.0/doc/api/ 0000755 0001751 0000200 00000000000 15001705403 013375 5 ustar 00runner docker ././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/api/Makefile 0000644 0001751 0000200 00000001345 15001705132 015037 0 ustar 00runner docker # Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
html:
@$(SPHINXBUILD) -b html "$(SOURCEDIR)" -d "$(BUILDDIR)/doctrees" ../html/apidoc $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/api/conf.py 0000644 0001751 0000200 00000002754 15001705132 014703 0 ustar 00runner docker import os
import sys
sys.path.insert(0, os.path.abspath('../../src'))
from lxml import __version__ as lxml_version
# -- Project information -----------------------------------------------------
project = 'lxml'
copyright = '2020, lxml dev team'
author = 'lxml dev team'
version = lxml_version
# -- General configuration ---------------------------------------------------
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.viewcode',
'sphinx_rtd_theme',
]
language = 'en'
exclude_patterns = ['_build']
# -- Options for HTML output -------------------------------------------------
html_theme = 'sphinx_rtd_theme'
html_logo = '../html/python-xml.png'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
#html_static_path = ['_static']
html_theme_options = {
'collapse_navigation': False,
'titles_only': True,
}
# -- Extension configuration -------------------------------------------------
autodoc_default_options = {
'ignore-module-all': True,
'private-members': True,
'inherited-members': True,
}
autodoc_member_order = 'groupwise'
intersphinx_mapping = {
"lxml": ("https://lxml.de/apidoc/", None),
}
# -- Options for todo extension ----------------------------------------------
# If true, `todo` and `todoList` produce output, else they produce nothing.
#todo_include_todos = True
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/api.txt 0000644 0001751 0000200 00000052371 15001705132 014145 0 ustar 00runner docker ===========================
APIs specific to lxml.etree
===========================
lxml.etree tries to follow established APIs wherever possible. Sometimes,
however, the need to expose a feature in an easy way led to the invention of a
new API. This page describes the major differences and a few additions to the
main ElementTree API.
For a complete reference of the API, see the `generated API
documentation`_.
Separate pages describe the support for `parsing XML`_, executing `XPath and
XSLT`_, `validating XML`_ and interfacing with other XML tools through the
`SAX-API`_.
lxml is extremely extensible through `XPath functions in Python`_, custom
`Python element classes`_, custom `URL resolvers`_ and even `at the C-level`_.
.. _`parsing XML`: parsing.html
.. _`XPath and XSLT`: xpathxslt.html
.. _`validating XML`: validation.html
.. _`SAX-API`: sax.html
.. _`XPath functions in Python`: extensions.html
.. _`Python element classes`: element_classes.html
.. _`at the C-level`: capi.html
.. _`URL resolvers`: resolvers.html
.. _`generated API documentation`: api/index.html
.. contents::
..
1 lxml.etree
2 Other Element APIs
3 Trees and Documents
4 Iteration
5 Error handling on exceptions
6 Error logging
7 Serialisation
8 Incremental XML generation
9 CDATA
10 XInclude and ElementInclude
..
>>> from io import BytesIO
>>> def StringIO(s=None):
... if isinstance(s, str): s = s.encode("UTF-8")
... return BytesIO(s)
lxml.etree
----------
lxml.etree tries to follow the `ElementTree API`_ wherever it can. There are
however some incompatibilities (see `compatibility`_). The extensions are
documented here.
.. _`ElementTree API`: http://effbot.org/zone/element-index.htm
.. _`compatibility`: compatibility.html
If you need to know which version of lxml is installed, you can access the
``lxml.etree.LXML_VERSION`` attribute to retrieve a version tuple. Note,
however, that it did not exist before version 1.0, so you will get an
AttributeError in older versions. The versions of libxml2 and libxslt are
available through the attributes ``LIBXML_VERSION`` and ``LIBXSLT_VERSION``.
The following examples usually assume this to be executed first:
.. sourcecode:: pycon
>>> from lxml import etree
..
>>> import sys
>>> from lxml import etree as _etree
>>> if sys.version_info[0] >= 3:
... class etree_mock(object):
... def __getattr__(self, name): return getattr(_etree, name)
... def tostring(self, *args, **kwargs):
... s = _etree.tostring(*args, **kwargs)
... if isinstance(s, bytes) and bytes([10]) in s: s = s.decode("utf-8") # CR
... if s[-1] == '\n': s = s[:-1]
... return s
... else:
... class etree_mock(object):
... def __getattr__(self, name): return getattr(_etree, name)
... def tostring(self, *args, **kwargs):
... s = _etree.tostring(*args, **kwargs)
... if s[-1] == '\n': s = s[:-1]
... return s
>>> etree = etree_mock()
Other Element APIs
------------------
While lxml.etree itself uses the ElementTree API, it is possible to replace
the Element implementation by `custom element subclasses`_. This has been
used to implement well-known XML APIs on top of lxml. For example, lxml ships
with a data-binding implementation called `objectify`_, which is similar to
the `Amara bindery`_ tool.
lxml.etree comes with a number of `different lookup schemes`_ to customize the
mapping between libxml2 nodes and the Element classes used by lxml.etree.
.. _`custom element subclasses`: element_classes.html
.. _`objectify`: objectify.html
.. _`different lookup schemes`: element_classes.html#setting-up-a-class-lookup-scheme
.. _`Amara bindery`: http://uche.ogbuji.net/tech/4suite/amara/
Trees and Documents
-------------------
Compared to the original ElementTree API, lxml.etree has an extended tree
model. It knows about parents and siblings of elements:
.. sourcecode:: pycon
>>> root = etree.Element("root")
>>> a = etree.SubElement(root, "a")
>>> b = etree.SubElement(root, "b")
>>> c = etree.SubElement(root, "c")
>>> d = etree.SubElement(root, "d")
>>> e = etree.SubElement(d, "e")
>>> b.getparent() == root
True
>>> print(b.getnext().tag)
c
>>> print(c.getprevious().tag)
b
Elements always live within a document context in lxml. This implies that
there is also a notion of an absolute document root. You can retrieve an
ElementTree for the root node of a document from any of its elements.
.. sourcecode:: pycon
>>> tree = d.getroottree()
>>> print(tree.getroot().tag)
root
Note that this is different from wrapping an Element in an ElementTree. You
can use ElementTrees to create XML trees with an explicit root node:
.. sourcecode:: pycon
>>> tree = etree.ElementTree(d)
>>> print(tree.getroot().tag)
d
>>> etree.tostring(tree)
b'
>>> from collections import deque
>>> queue = deque([root])
>>> while queue:
... el = queue.popleft() # pop next element
... queue.extend(el) # append its children
... print(el.tag)
root
a
d
b
c
e
See also the section on the utility functions ``iterparse()`` and
``iterwalk()`` in the `parser documentation`_.
.. _`parser documentation`: parsing.html#iterparse-and-iterwalk
Error handling on exceptions
----------------------------
Libxml2 provides error messages for failures, be it during parsing, XPath
evaluation or schema validation. The preferred way of accessing them is
through the local ``error_log`` property of the respective evaluator or
transformer object. See their documentation for details.
However, lxml also keeps a global error log of all errors that occurred at the
application level. Whenever an exception is raised, you can retrieve the
errors that occurred and "might have" lead to the problem from the error log
copy attached to the exception:
.. sourcecode:: pycon
>>> etree.clear_error_log()
>>> broken_xml = '''
...
...
...
... '''
>>> try:
... etree.parse(StringIO(broken_xml))
... except etree.XMLSyntaxError as e:
... pass # just put the exception into e
..
>>> etree.clear_error_log()
>>> try:
... etree.parse(StringIO(broken_xml))
... except etree.XMLSyntaxError:
... import sys; e = sys.exc_info()[1]
Once you have caught this exception, you can access its ``error_log`` property
to retrieve the log entries or filter them by a specific type, error domain or
error level:
.. sourcecode:: pycon
>>> log = e.error_log.filter_from_level(etree.ErrorLevels.FATAL)
>>> print(log[0])
:4:8:FATAL:PARSER:ERR_TAG_NAME_MISMATCH: Opening and ending tag mismatch: a line 3 and root
This might look a little cryptic at first, but it is the information that
libxml2 gives you. At least the message at the end should give you a hint
what went wrong and you can see that the fatal errors (FATAL) happened during
parsing (PARSER) lines 4, column 8 and line 5, column 1 of a string (,
or the filename if available). Here, PARSER is the so-called error domain,
see ``lxml.etree.ErrorDomains`` for that. You can get it from a log entry
like this:
.. sourcecode:: pycon
>>> entry = log[0]
>>> print(entry.domain_name)
PARSER
>>> print(entry.type_name)
ERR_TAG_NAME_MISMATCH
>>> print(entry.filename)
There is also a convenience attribute ``error_log.last_error`` that returns the
last error or fatal error that occurred, so that it's easy to test if there was
an error at all. Note, however, that there might have been more than one error,
and the first error that occurred might be more relevant in some cases.
Error logging
-------------
lxml.etree supports logging libxml2 messages to the Python stdlib logging
module. This is done through the ``etree.PyErrorLog`` class. It disables the
error reporting from exceptions and forwards log messages to a Python logger.
To use it, see the descriptions of the function ``etree.useGlobalPythonLog``
and the class ``etree.PyErrorLog`` for help. Note that this does not affect
the local error logs of XSLT, XMLSchema, etc.
Serialisation
-------------
C14N
....
lxml.etree has support for `C14N 1.0
Note the newline that is appended at the end when pretty printing the
output. It was added in lxml 2.0.
XML declaration
...............
By default, lxml (just as ElementTree) outputs the XML declaration only if it
is required by the standard:
.. sourcecode:: pycon
>>> unicode_root = etree.Element( "t\u3120st" )
>>> unicode_root.text = "t\u0A0Ast"
>>> etree.tostring(unicode_root, encoding="utf-8")
b't\xe0\xa8\x8ast '
>>> print(etree.tostring(unicode_root, encoding="iso-8859-1"))
tਊst
Also see the general remarks on `Unicode support`_.
.. _`Unicode support`: parsing.html#python-unicode-strings
You can enable or disable the declaration explicitly by passing another
keyword argument for the serialisation:
.. sourcecode:: pycon
>>> print(etree.tostring(root, xml_declaration=True))
text
``xmlfile()`` accepts a file path as first argument, or a file(-like)
object, as in the example above. In the first case, it takes care to
open and close the file itself, whereas file(-like) objects are not
closed by default. This is left to the code that opened them. Since
lxml 3.4, however, you can pass the argument ``close=True`` to make
lxml call the object's ``.close()`` method when exiting the xmlfile
context manager.
To insert pre-constructed Elements and subtrees, just pass them
into ``write()``::
>>> f = BytesIO()
>>> with etree.xmlfile(f) as xf:
... with xf.element('abc'):
... with xf.element('in'):
...
... for value in '123':
... # construct a really complex XML tree
... el = etree.Element('xyz', attr=value)
...
... xf.write(el)
...
... # no longer needed, discard it right away!
... el = None
>>> print(f.getvalue().decode('utf-8'))
`_::
def writer(out_stream):
with xmlfile(out_stream) as xf:
with xf.element('{http://etherx.jabber.org/streams}stream'):
while True:
el = (yield)
xf.write(el)
xf.flush()
w = writer(stream)
next(w) # start writing (run up to 'yield')
Then, whenever XML elements are available for writing, call
::
w.send(element)
And when done::
w.close()
Note the additional ``xf.flush()`` call in the example above, which is
available since lxml 3.4. Normally, the output stream is buffered to
avoid excessive I/O calls. Whenever the internal buffer fills up, its
content is written out. In the case above, however, we want to make
sure that each message that we write (i.e. each element subtree) is
written out immediately, so we flush the content explicitly at the
right point.
Alternatively, if buffering is not desired at all, it can be disabled
by passing the flag ``buffered=False`` into ``xmlfile()`` (also since
lxml 3.4).
Here is a similar example using an async coroutine in Py3.5 or later, which is
supported since lxml 4.0. The output stream is expected to have methods
``async def write(self, data)`` and ``async def close(self)`` in this case.
::
async def writer(out_stream, xml_messages):
async with xmlfile(out_stream) as xf:
async with xf.element('{http://etherx.jabber.org/streams}stream'):
async for el in xml_messages:
await xf.write(el)
await xf.flush()
class DummyAsyncOut(object):
async def write(self, data):
print(data.decode('utf8'))
async def close(self):
pass
stream = DummyAsyncOut()
async_writer = writer(stream, async_message_stream)
CDATA
-----
By default, lxml's parser will strip CDATA sections from the tree and
replace them by their plain text content. As real applications for
CDATA are rare, this is the best way to deal with this issue.
However, in some cases, keeping CDATA sections or creating them in a
document is required to adhere to existing XML language definitions.
For these special cases, you can instruct the parser to leave CDATA
sections in the document:
.. sourcecode:: pycon
>>> parser = etree.XMLParser(strip_cdata=False)
>>> root = etree.XML('test '
>>> root.text = etree.CDATA(root.text)
>>> root.text
'test'
>>> etree.tostring(root)
b'
... ''')
>>> tree = etree.parse(data)
>>> tree.xinclude()
>>> print(etree.tostring(tree.getroot()))
Note that the ElementTree compatible ElementInclude_ module is also supported
as ``lxml.ElementInclude``. It has the additional advantage of supporting
custom `URL resolvers`_ at the Python level. The normal XInclude mechanism
cannot deploy these. If you need ElementTree compatibility or custom
resolvers, you have to stick to the external Python module.
.. _ElementInclude: http://effbot.org/zone/element-xinclude.htm
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/build.txt 0000644 0001751 0000200 00000030012 15001705132 014457 0 ustar 00runner docker How to build lxml from source
=============================
To build lxml from source, you need libxml2 and libxslt properly
installed, *including the header files*. These are likely shipped in
separate ``-dev`` or ``-devel`` packages like ``libxml2-dev``, which
you must install before trying to build lxml.
.. contents::
..
1 Cython
2 Github, git and hg
3 Building the sources
4 Running the tests and reporting errors
5 Building an egg
6 Building lxml on MacOS-X
7 Static linking on Windows
8 Building Debian packages from SVN sources
Cython
------
.. _pip: http://pypi.python.org/pypi/pip
.. _Cython: http://cython.org
.. _wheel: https://wheel.readthedocs.io/en/latest/
The lxml.etree and lxml.objectify modules are written in Cython_.
Since we distribute the Cython-generated .c files with lxml releases,
however, you do not need Cython to build lxml from the normal release
sources. We even encourage you to *not install Cython* for a normal
release build, as the generated C code can vary quite heavily between
Cython versions, which may or may not generate correct code for lxml.
The pre-generated release sources were tested and therefore are known
to work.
So, if you want a reliable build of lxml, we suggest to a) use a
source release of lxml and b) disable or uninstall Cython for the
build.
*Only* if you are interested in building lxml from a checkout of the
developer sources (e.g. to test a bug fix that has not been release
yet) or if you want to be an lxml developer, then you do need a
working Cython installation. You can use pip_ to install it::
pip install -r requirements.txt
https://github.com/lxml/lxml/blob/master/requirements.txt
lxml 5.x currently uses at least Cython 3.0. Later release versions
are generally preferred. lxml 4.x used Cython 0.29.x instead.
Github, git and hg
-------------------
The lxml package is developed in a repository on Github_ using
Mercurial_ and the `hg-git`_ plugin. You can retrieve the current
developer version using::
hg clone git+ssh://git@github.com/lxml/lxml.git lxml
Or, using git::
git clone ssh://git@github.com/lxml/lxml.git lxml
This will create a directory ``lxml`` and download the source into it,
including the complete development history. Don't be afraid, the
repository download is fairly quick. You can also browse the
`lxml repository`_ through the web or download a ZIP archive with the
`latest master branch `_.
.. _Github: https://github.com/lxml/
.. _Mercurial: http://mercurial.selenic.com/
.. _`hg-git`: http://hg-git.github.com/
.. _`lxml repository`: https://github.com/lxml/lxml
.. _`source tar-ball`: https://github.com/lxml/lxml/tarball/master
Building the sources
---------------------
Clone the source repository as described above (or download
the `source tar-ball`_ and unpack it) and then type::
python setup.py build
or::
python setup.py bdist_egg # requires 'setuptools' or 'distribute'
To (re-)build the C sources with Cython, you must additionally pass the
option ``--with-cython``::
python setup.py build --with-cython
If you want to test lxml from the source directory, it is better to build it
in-place like this::
python setup.py build_ext -i --with-cython
or, in Unix-like environments::
make inplace
To speed up the build in test environments (e.g. on a continuous
integration server), set the ``CFLAGS`` environment variable to
disable C compiler optimisations (e.g. "-O0" for gcc, that's
minus-oh-zero), for example::
CFLAGS="-O0" make inplace
If you get errors about missing header files (e.g. ``Python.h`` or
``libxml/xmlversion.h``) then you need to make sure the development
packages of Python, libxml2 and libxslt are properly installed. On
Linux distributions, they are usually called something like
``libxml2-dev`` or ``libxslt-devel``. If these packages were
installed in non-standard places, try passing the following option to
setup.py to make sure the right config is found::
python setup.py build --with-xslt-config=/path/to/xslt-config
There are also env vars to allow overriding the config tool::
env XML2_CONFIG=/path/to/xml2-config python build
You may also use ``pkg-config`` as the tools::
env XSLT_CONFIG="pkg-config libxslt" python setup.py build
If this doesn't help, you may have to add the location of the header
files to the include path like::
python setup.py build_ext -i -I /usr/include/libxml2
where the file is in ``/usr/include/libxml2/libxml/xmlversion.h``
For static builds, if you get an error saying "recompile with -fPIC",
do so by adding it to your `CFLAGS` environment variable:
``env CFLAGS="$CFLAGS -fPIC"``, such as::
env CFLAGS="$CFLAGS -fPIC" python3 setup.py build_ext -i --with-cython --static-deps
To use lxml.etree in-place, you can place lxml's ``src`` directory
on your Python module search path (PYTHONPATH) and then import
``lxml.etree`` to play with it::
# cd lxml
# PYTHONPATH=src python3
Python 3.10.2
Type "help", "copyright", "credits" or "license" for more information.
>>> from lxml import etree
>>>
For non-static builds, you may have to set ``LD_LIBRARY_PATH`` to where the
shared object files for libxml2 and libxslt are, such as ``/usr/local/lib``. For
example::
PYTHONPATH=src LD_LIBRARY_PATH=/usr/local/lib python3
To make sure everything gets recompiled cleanly after changes, you can
run ``make clean`` or delete the file ``src/lxml/etree.c``.
Running the tests and reporting errors
--------------------------------------
The source distribution (tgz) and the source repository contain a test
suite for lxml. You can run it from the top-level directory::
python test.py
Note that the test script only tests the in-place build (see distutils
building above), as it searches the ``src`` directory. You can use the
following one-step command to trigger an in-place build and test it::
make test
This also runs the ElementTree and cElementTree compatibility tests. To call
them separately, make sure you have lxml on your PYTHONPATH first, then run::
python selftest.py
and::
python selftest2.py
If the tests give failures, errors, or worse, segmentation faults, we'd really
like to know. Please contact us on the `mailing list`_, and please specify
the version of lxml, libxml2, libxslt and Python you were using, as well as
your operating system type (Linux, Windows, MacOS-X, ...).
.. _`mailing list`: https://lxml.de/mailinglist/
Building an egg or wheel
------------------------
This is the procedure to make an lxml egg or wheel_ for your platform.
It assumes that you have ``setuptools`` or ``distribute`` installed, as well
as the ``wheel`` package.
First, download the lxml-x.y.tar.gz release. This contains the pregenerated
C files so that you can be sure you build exactly from the release sources.
Unpack them and ``cd`` into the resulting directory. Then, to build a wheel,
simply run the command
::
python setup.py bdist_wheel
or, to build a statically linked wheel with all of libxml2, libxslt and
friends compiled in, run
python setup.py bdist_wheel --static-deps
The resulting .whl file will be written into the ``dist`` directory.
To build an egg file, run
::
python setup.py build_egg
If you are on a Unix-like platform, you can first build the extension modules
using
::
python setup.py build
and then ``cd`` into the directory ``build/lib.your.platform`` to call
``strip`` on any ``.so`` file you find there. This reduces the size of
the binary distribution considerably. Then, from the package root directory,
call
::
python setup.py bdist_egg
This will quickly package the pre-built packages into an egg file and
drop it into the ``dist`` directory.
Building lxml on MacOS-X
------------------------
Apple regularly ships new system releases with horribly outdated
system libraries. This is specifically the case for libxml2 and
libxslt, where the system provided versions used to be too old
to even build lxml for a long time.
While the Unix environment in MacOS-X makes it relatively easy to
install Unix/Linux style package management tools and new software, it
actually seems to be hard to get libraries set up for exclusive usage
that MacOS-X ships in an older version. Alternative distributions
(like macports) install their libraries in addition to the system
libraries, but the compiler and the runtime loader on MacOS still sees
the system libraries before the new libraries. This can lead to
undebuggable crashes where the newer library seems to be loaded but
the older system library is used.
Apple discourages static building against libraries, which would help
working around this problem. Apple does not ship static library
binaries with its system and several package management systems follow
this decision. Therefore, building static binaries requires building
the dependencies first. The ``setup.py`` script does this
automatically when you call it like this::
python setup.py build --static-deps
This will download and build the latest versions of libxml2 and
libxslt from the official FTP download site. If you want to use
specific versions, or want to prevent any online access, you can
download both ``tar.gz`` release files yourself, place them into a
subdirectory ``libs`` in the lxml distribution, and call ``setup.py``
with the desired target versions like this::
python setup.py build --static-deps \
--libxml2-version=2.9.12 \
--libxslt-version=1.1.34 \
sudo python setup.py install
Instead of ``build``, you can use any target, like ``bdist_egg``
if you want to use setuptools to build an installable egg, or
``bdist_wheel`` for a wheel package.
Note that this also works with pip_. Since you can't pass
command line options in this case, you have to use an environment
variable instead::
STATIC_DEPS=true pip install lxml
To install the package into the system Python package directory,
run the installation with "sudo"::
STATIC_DEPS=true sudo pip install lxml
The ``STATICBUILD`` environment variable is handled equivalently to
the ``STATIC_DEPS`` variable, but is used by some other extension
packages, too.
If you decide to do a non-static build, you may also have to install
the command line tools in addition to the XCode build environment.
They are available as a restricted download from here:
https://developer.apple.com/downloads/index.action?=command%20line%20tools#
Without them, the compiler may not find the necessary header files of
the XML libraries, according to the second comment in this ticket:
https://bugs.launchpad.net/lxml/+bug/1244094
Static linking on Windows
-------------------------
Most operating systems have proper package management that makes installing
current versions of libxml2 and libxslt easy. The most famous exception is
Microsoft Windows, which entirely lacks these capabilities. To work around
the limits of this platform, lxml's installation can download pre-built
packages of the dependencies and build statically against them. Assuming
you have a proper C compiler setup to build Python extensions, this should
work::
python setup.py bdist_wininst --static-deps
It should create a windows installer in the ``pkg`` directory.
Building Debian packages from SVN sources
-----------------------------------------
`Andreas Pakulat`_ proposed the following approach.
.. _`Andreas Pakulat`: http://thread.gmane.org/gmane.comp.python.lxml.devel/1239/focus=1249
* ``apt-get source lxml``
* remove the unpacked directory
* tar.gz the lxml SVN version and replace the orig.tar.gz that lies in the
directory
* check md5sum of created tar.gz file and place new sum and size in dsc file
* do ``dpkg-source -x lxml-[VERSION].dsc`` and cd into the newly created directory
* run ``dch -i`` and add a comment like "use trunk version", this will
increase the debian version number so apt/dpkg won't get confused
* run ``dpkg-buildpackage -rfakeroot -us -uc`` to build the package
In case ``dpkg-buildpackage`` tells you that some dependencies are missing, you
can either install them manually or run ``apt-get build-dep lxml``.
That will give you .deb packages in the parent directory which can be
installed using ``dpkg -i``.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/capi.txt 0000644 0001751 0000200 00000007722 15001705132 014310 0 ustar 00runner docker ==============================
The public C-API of lxml.etree
==============================
As of version 1.1, lxml.etree provides a public C-API. This allows external
C extensions to efficiently access public functions and classes of lxml,
without going through the Python API.
The API is described in the file `etreepublic.pxd`_, which is directly
c-importable by extension modules implemented in Cython_.
.. _`etreepublic.pxd`: https://github.com/lxml/lxml/blob/master/src/lxml/includes/etreepublic.pxd
.. _Cython: https://cython.org
.. contents::
..
1 Passing generated trees through Python
2 Writing external modules in Cython
3 Writing external modules in C
Passing generated trees through Python
--------------------------------------
This is the most simple way to integrate with lxml. It does not require
any C-level integration but uses a Python function to wrap an externally
generated libxml2 document in lxml.
The external module that creates the libxml2 tree must pack the document
pointer into a `PyCapsule `_
object. This can then be passed into lxml with the function
``lxml.etree.adopt_external_document()``. It also takes an optional lxml
parser instance to associate with the document, in order to configure the
Element class lookup, relative URL lookups, etc.
See the `API reference `_
for further details.
The same functionality is available as part of the public C-API in form
of the C function ``adoptExternalDocument()``.
Writing external modules in Cython
----------------------------------
This is the easiest way of extending lxml at the C level. A Cython_
module should start like this::
# My Cython extension
# directive pointing compiler to lxml header files;
# use ``aliases={"LXML_PACKAGE_DIR": lxml.__path__}``
# argument to cythonize in setup.py to dynamically
# determine dir at compile time
# distutils: include_dirs = LXML_PACKAGE_DIR
# import the public functions and classes of lxml.etree
cimport lxml.includes.etreepublic as cetree
# import the lxml.etree module in Python
cdef object etree
from lxml import etree
# initialize the access to the C-API of lxml.etree
cetree.import_lxml__etree()
From this line on, you can access all public functions of lxml.etree
from the ``cetree`` namespace like this::
# build a tag name from namespace and element name
py_tag = cetree.namespacedNameFromNsName("http://some/url", "myelement")
Public lxml classes are easily subclassed. For example, to implement
and set a new default element class, you can write Cython code like
the following::
from lxml.includes.etreepublic cimport ElementBase
cdef class NewElementClass(ElementBase):
def set_value(self, myval):
self.set("my_attribute", myval)
etree.set_element_class_lookup(
etree.ElementDefaultClassLookup(element=NewElementClass))
Writing external modules in C
-----------------------------
If you really feel like it, you can also interface with lxml.etree straight
from C code. All you have to do is include the header file for the public
API, import the ``lxml.etree`` module and then call the import function:
.. sourcecode:: c
/* My C extension */
/* common includes */
#include "Python.h"
#include "stdio.h"
#include "string.h"
#include "stdarg.h"
#include "libxml/xmlversion.h"
#include "libxml/encoding.h"
#include "libxml/hash.h"
#include "libxml/tree.h"
#include "libxml/xmlIO.h"
#include "libxml/xmlsave.h"
#include "libxml/globals.h"
#include "libxml/xmlstring.h"
/* lxml.etree specific includes */
#include "lxml-version.h"
#include "etree_defs.h"
#include "etree.h"
/* setup code */
import_lxml__etree()
Note that including ``etree.h`` does not automatically include the
header files it requires. Note also that the above list of common
includes may not be sufficient.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/compatibility.txt 0000644 0001751 0000200 00000021424 15001705132 016240 0 ustar 00runner docker =======================================
ElementTree compatibility of lxml.etree
=======================================
A lot of care has been taken to ensure compatibility between etree and
ElementTree. Nonetheless, some differences and incompatibilities exist:
* Importing etree is obviously different; etree uses a lower-case
package name, while ElementTree uses a combination of upper-case and
lower case in imports:
.. sourcecode:: python
# etree
from lxml.etree import Element
# ElementTree
from elementtree.ElementTree import Element
# ElementTree in the Python 2.5 standard library
from xml.etree.ElementTree import Element
When switching over code from ElementTree to lxml.etree, and you're using
the package name prefix 'ElementTree', you can do the following:
.. sourcecode:: python
# instead of
from elementtree import ElementTree
# use
from lxml import etree as ElementTree
* lxml.etree offers a lot more functionality, such as XPath, XSLT, Relax NG,
and XML Schema support, which (c)ElementTree does not offer.
* etree has a different idea about Python unicode strings than ElementTree.
In most parts of the API, ElementTree uses plain strings and unicode strings
as what they are. This includes Element.text, Element.tail and many other
places. However, the ElementTree parsers assume by default that any string
(`str` or `unicode`) contains ASCII data. They raise an exception if
strings do not match the expected encoding.
etree has the same idea about plain strings (`str`) as ElementTree. For
unicode strings, however, etree assumes throughout the API that they are
Python unicode encoded strings rather than byte data. This includes the
parsers. It is therefore perfectly correct to pass XML unicode data into
the etree parsers in form of Python unicode strings. It is an error, on the
other hand, if unicode strings specify an encoding in their XML declaration,
as this conflicts with the characteristic encoding of Python unicode
strings.
* ElementTree allows you to place an Element in two different trees at the
same time. Thus, this:
.. sourcecode:: python
a = Element('a')
b = SubElement(a, 'b')
c = Element('c')
c.append(b)
will result in the following tree a:
.. sourcecode:: xml
>>> sel.css
'div.content'
The selector actually compiles to XPath, and you can see the
expression by inspecting the object:
.. sourcecode:: pycon
>>> sel.path
"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' content ')]"
To use the selector, simply call it with a document or element
object:
.. sourcecode:: pycon
>>> from lxml.etree import fromstring
>>> h = fromstring('''''')
>>> [e.get('id') for e in sel(h)]
['inner']
Using ``CSSSelector`` is equivalent to translating with ``cssselect``
and using the ``XPath`` class:
.. sourcecode:: pycon
>>> from cssselect import GenericTranslator
>>> from lxml.etree import XPath
>>> sel = XPath(GenericTranslator().css_to_xpath('div.content'))
``CSSSelector`` takes a ``translator`` parameter to let you choose which
translator to use. It can be ``'xml'`` (the default), ``'xhtml'``, ``'html'``
or a `Translator object`_.
.. _Translator object: http://packages.python.org/cssselect/#cssselect.GenericTranslator
The cssselect method
====================
lxml ``Element`` objects have a ``cssselect`` convenience method.
.. sourcecode:: pycon
>>> h.cssselect('div.content') == sel(h)
True
Note however that pre-compiling the expression with the ``CSSSelector`` or
``XPath`` class can provide a substantial speedup.
The method also accepts a ``translator`` parameter. On ``HtmlElement``
objects, the default is changed to ``'html'``.
Supported Selectors
===================
Most `Level 3`_ selectors are supported. The details are in the
`cssselect documentation`_.
.. _Level 3: http://www.w3.org/TR/2011/REC-css3-selectors-20110929/
.. _cssselect documentation: http://packages.python.org/cssselect/#supported-selectors
Namespaces
==========
In CSS you can use ``namespace-prefix|element``, similar to
``namespace-prefix:element`` in an XPath expression. In fact, it maps
one-to-one, and the same rules are used to map namespace prefixes to
namespace URIs: the ``CSSSelector`` class accepts a dictionary as its
``namespaces`` argument.
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/docstructure.py 0000644 0001751 0000200 00000002125 15001705132 015723 0 ustar 00runner docker
import os
if os.path.exists(os.path.join(os.path.dirname(__file__), '..', 'funding.txt')):
funding = ('../funding.txt',)
else:
funding = ()
SITE_STRUCTURE = [
('lxml', ('main.txt', 'intro.txt', '../INSTALL.txt', # 'lxml2.txt',
'performance.txt', 'compatibility.txt', 'FAQ.txt') + funding),
('Developing with lxml', ('tutorial.txt', '@API reference',
'api.txt', 'parsing.txt',
'validation.txt', 'xpathxslt.txt',
'objectify.txt', 'lxmlhtml.txt',
'cssselect.txt', 'elementsoup.txt',
'html5parser.txt')),
('Extending lxml', ('resolvers.txt', 'extensions.txt',
'element_classes.txt', 'sax.txt', 'capi.txt')),
('Developing lxml', ('build.txt', 'lxml-source-howto.txt',
'@Release Changelog', '../CREDITS.txt')),
]
HREF_MAP = {
"API reference" : "apidoc/lxml.html"
}
BASENAME_MAP = {
'main' : 'index',
'INSTALL' : 'installation',
'CREDITS' : 'credits',
}
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/element_classes.txt 0000644 0001751 0000200 00000051407 15001705132 016541 0 ustar 00runner docker ====================================
Using custom Element classes in lxml
====================================
lxml has very sophisticated support for custom Element classes. You
can provide your own classes for Elements and have lxml use them by
default for all elements generated by a specific parser, only for a
specific tag name in a specific namespace or even for an exact element
at a specific position in the tree.
Custom Elements must inherit from the ``lxml.etree.ElementBase`` class, which
provides the Element interface for subclasses:
.. sourcecode:: pycon
>>> from lxml import etree
>>> class honk(etree.ElementBase):
... @property
... def honking(self):
... return self.get('honking') == 'true'
This defines a new Element class ``honk`` with a property ``honking``.
The following document describes how you can make lxml.etree use these
custom Element classes.
.. contents::
..
1 Background on Element proxies
2 Element initialization
3 Setting up a class lookup scheme
3.1 Default class lookup
3.2 Namespace class lookup
3.3 Attribute based lookup
3.4 Custom element class lookup
3.5 Tree based element class lookup in Python
4 Generating XML with custom classes
5 Implementing namespaces
Background on Element proxies
=============================
Being based on libxml2, lxml.etree holds the entire XML tree in a C
structure. To communicate with Python code, it creates Python proxy
objects for the XML elements on demand.
.. image:: proxies.png
The mapping between C elements and Python Element classes is
completely configurable. When you ask lxml.etree for an Element by
using its API, it will instantiate your classes for you. All you have
to do is tell lxml which class to use for which kind of Element. This
is done through a class lookup scheme, as described in the sections
below.
Element initialization
======================
There is one thing to know up front. Element classes *must not* have
an ``__init___`` or ``__new__`` method. There should not be any
internal state either, except for the data stored in the underlying
XML tree. Element instances are created and garbage collected at
need, so there is normally no way to predict when and how often a
proxy is created for them. Even worse, when the ``__init__`` method
is called, the object is not even initialized yet to represent the XML
tag, so there is not much use in providing an ``__init__`` method in
subclasses.
Most use cases will not require any class initialisation or proxy
state, so you can content yourself with skipping to the next section
for now. However, if you really need to set up your element class on
instantiation, or need a way to persistently store state in the proxy
instances instead of the XML tree, here is a way to do so.
There is one important guarantee regarding Element proxies. Once a
proxy has been instantiated, it will keep alive as long as there is a
Python reference to it, and any access to the XML element in the tree
will return this very instance. Therefore, if you need to store local
state in a custom Element class (which is generally discouraged), you
can do so by keeping the Elements in a tree alive. If the tree
doesn't change, you can simply do this:
.. sourcecode:: python
proxy_cache = list(root.iter())
or
.. sourcecode:: python
proxy_cache = set(root.iter())
or use any other suitable container. Note that you have to keep this
cache manually up to date if the tree changes, which can get tricky in
cases.
For proxy initialisation, ElementBase classes have an ``_init()``
method that can be overridden, as oppose to the normal ``__init__()``
method. It can be used to modify the XML tree, e.g. to construct
special children or verify and update attributes.
The semantics of ``_init()`` are as follows:
* It is called once on Element class instantiation time. That is,
when a Python representation of the element is created by lxml. At
that time, the element object is completely initialized to represent
a specific XML element within the tree.
* The method has complete access to the XML tree. Modifications can be done
in exactly the same way as anywhere else in the program.
* Python representations of elements may be created multiple times during the
lifetime of an XML element in the underlying C tree. The ``_init()`` code
provided by subclasses must take special care by itself that multiple
executions either are harmless or that they are prevented by some kind of
flag in the XML tree. The latter can be achieved by modifying an attribute
value or by removing or adding a specific child node and then verifying this
before running through the init process.
* Any exceptions raised in ``_init()`` will be propagated through the API
call that lead to the creation of the Element. So be careful with the code
you write here as its exceptions may turn up in various unexpected places.
Setting up a class lookup scheme
================================
The first thing to do when deploying custom element classes is to register a
class lookup scheme on a parser. lxml.etree provides quite a number of
different schemes that also support class lookup based on namespaces or
attribute values. Most lookups support fallback chaining, which allows the
next lookup mechanism to take over when the previous one fails to find a
class.
For example, setting the ``honk`` Element as a default element class
for a parser works as follows:
.. sourcecode:: pycon
>>> parser_lookup = etree.ElementDefaultClassLookup(element=honk)
>>> parser = etree.XMLParser()
>>> parser.set_element_class_lookup(parser_lookup)
There is one drawback of the parser based scheme: the ``Element()`` factory
does not know about your specialised parser and creates a new document that
deploys the default parser:
.. sourcecode:: pycon
>>> el = etree.Element("root")
>>> print(isinstance(el, honk))
False
You should therefore avoid using this factory function in code that
uses custom classes. The ``makeelement()`` method of parsers provides
a simple replacement:
.. sourcecode:: pycon
>>> el = parser.makeelement("root")
>>> print(isinstance(el, honk))
True
If you use a parser at the module level, you can easily redirect a module
level ``Element()`` factory to the parser method by adding code like this:
.. sourcecode:: pycon
>>> module_level_parser = etree.XMLParser()
>>> Element = module_level_parser.makeelement
While the ``XML()`` and ``HTML()`` factories also depend on the default
parser, you can pass them a different parser as second argument:
.. sourcecode:: pycon
>>> element = etree.XML("some
Implementing namespaces
=======================
lxml allows you to implement namespaces, in a rather literal sense. After
setting up the namespace class lookup mechanism as described above, you can
build a new element namespace (or retrieve an existing one) by calling the
``get_namespace(uri)`` method of the lookup:
.. sourcecode:: pycon
>>> lookup = etree.ElementNamespaceClassLookup()
>>> parser = etree.XMLParser()
>>> parser.set_element_class_lookup(lookup)
>>> namespace = lookup.get_namespace('http://hui.de/honk')
and then register the new element type with that namespace, say, under the tag
name ``honk``:
.. sourcecode:: pycon
>>> namespace['honk'] = honk
If you have many Element classes declared in one module, and they are
all named like the elements they create, you can simply use
``namespace.update(globals())`` at the end of your module to declare them
automatically. The implementation is smart enough to ignore
everything that is not an Element class.
After this, you create and use your XML elements through the normal API of
lxml:
.. sourcecode:: pycon
>>> xml = ''
... ' ')
>>> honk_element = etree.XML(xml, parser)
>>> print(honk_element.honking)
True
>>> print(honk_element[0].honking)
Traceback (most recent call last):
...
AttributeError: 'lxml.etree._Element' object has no attribute 'honking'
>>> print(honk_element[1].text)
comment
You can therefore provide one implementation per element name in each
namespace and have lxml select the right one on the fly. If you want one
element implementation per namespace (ignoring the element name) or prefer
having a common class for most elements except a few, you can specify a
default implementation for an entire namespace by registering that class with
the empty element name (``None``).
You may consider following an object oriented approach here. If you build a
class hierarchy of element classes, you can also implement a base class for a
namespace that is used if no specific element class is provided. Again, you
can just pass None as an element name:
.. sourcecode:: pycon
>>> class HonkNSElement(etree.ElementBase):
... def honk(self):
... return "HONK"
>>> namespace[None] = HonkNSElement # default Element for namespace
>>> class HonkElement(HonkNSElement):
... @property
... def honking(self):
... return self.get('honking') == 'true'
>>> namespace['honk'] = HonkElement # Element for specific tag
Now you can rely on lxml to always return objects of type HonkNSElement or its
subclasses for elements of this namespace:
.. sourcecode:: pycon
>>> xml = (''
... ' ')
>>> honk_element = etree.fromstring(xml, parser)
>>> print(type(honk_element))
>>> print(type(honk_element[0]))
>>> print(honk_element.honking)
True
>>> print(honk_element.honk())
HONK
>>> print(honk_element[0].honk())
HONK
>>> print(honk_element[0].honking)
Traceback (most recent call last):
...
AttributeError: 'HonkNSElement' object has no attribute 'honking'
>>> print(honk_element[1].text) # uses fallback for non-elements
comment
Since lxml 4.1, the registration is more conveniently done with
class decorators. The namespace registry object is callable with
a name (or ``None``) as argument and can then be used as decorator.
.. sourcecode:: pycon
>>> honk_elements = lookup.get_namespace('http://hui.de/honk')
>>> @honk_elements(None)
... class HonkNSElement(etree.ElementBase):
... def honk(self):
... return "HONK"
If the class has the same name as the tag, you can also leave out the call
and use the blank decorator instead:
.. sourcecode:: pycon
>>> @honk_elements
... class honkel(HonkNSElement):
... @property
... def honking(self):
... return self.get('honking') == 'true'
>>> xml = '
>>> print(type(honk_element[0]))
././@PaxHeader 0000000 0000000 0000000 00000000026 00000000000 010213 x ustar 00 22 mtime=1745324634.0
lxml-5.4.0/doc/elementsoup.txt 0000644 0001751 0000200 00000017032 15001705132 015727 0 ustar 00runner docker ====================
BeautifulSoup Parser
====================
BeautifulSoup_ is a Python package for working with real-world and broken HTML,
just like `lxml.html `_. As of version 4.x, it can use
`different HTML parsers
`_,
each of which has its advantages and disadvantages (see the link).
lxml can make use of BeautifulSoup as a parser backend, just like BeautifulSoup
can employ lxml as a parser. When using BeautifulSoup from lxml, however, the
default is to use Python's integrated HTML parser in the
`html.parser `_ module.
In order to make use of the HTML5 parser of
`html5lib `_ instead, it is better
to go directly through the `html5parser module `_ in
``lxml.html``.
A very nice feature of BeautifulSoup is its excellent `support for encoding
detection`_ which can provide better results for real-world HTML pages that
do not (correctly) declare their encoding.
.. _BeautifulSoup: http://www.crummy.com/software/BeautifulSoup/
.. _`support for encoding detection`: http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicode-dammit
.. _ElementSoup: http://effbot.org/zone/element-soup.htm
lxml interfaces with BeautifulSoup through the ``lxml.html.soupparser``
module. It provides three main functions: ``fromstring()`` and ``parse()``
to parse a string or file using BeautifulSoup into an ``lxml.html``
document, and ``convert_tree()`` to convert an existing BeautifulSoup
tree into a list of top-level Elements.
.. contents::
..
1 Parsing with the soupparser
2 Entity handling
3 Using soupparser as a fallback
4 Using only the encoding detection
Parsing with the soupparser
===========================
The functions ``fromstring()`` and ``parse()`` behave as known from
lxml. The first returns a root Element, the latter returns an
ElementTree.
There is also a legacy module called ``lxml.html.ElementSoup``, which
mimics the interface provided by Fredrik Lundh's ElementSoup_
module. Note that the ``soupparser`` module was added in lxml 2.0.3.
Previous versions of lxml 2.0.x only have the ``ElementSoup`` module.
Here is a document full of tag soup, similar to, but not quite like, HTML:
.. sourcecode:: pycon
>>> tag_soup = '''
... Hello</head><body onload=crash()>Hi all<p>'''
All you need to do is pass it to the ``fromstring()`` function:
.. sourcecode:: pycon
>>> from lxml.html.soupparser import fromstring
>>> root = fromstring(tag_soup)
To see what we have here, you can serialise it:
.. sourcecode:: pycon
>>> from lxml.etree import tostring
>>> print(tostring(root, pretty_print=True).strip())
<html>
<meta/>
<head>
<title>Hello
Hi all