--- source_url: "https://pypi.org/project/trafilatura/0.0.2/?utm_source=openai" title: trafilatura · PyPI mirrored_at: 2026-08-20T03:01:42.508Z host: pypi.org cited_in_42a: true mirror_canonical: "https://index.42a.ai/pypi.org/project/trafilatura/0.0.2/index__q__utm_source_openai" --- > **Original source:** https://pypi.org/project/trafilatura/0.0.2/?utm_source=openai Code: [https://github.com/adbar/trafilatura](https://github.com/adbar/trafilatura) Issue tracker: [https://github.com/adbar/trafilatura/issues](https://github.com/adbar/trafilatura/issues) License: GNU GPL v3; see LICENSE file Robust extraction of main text content and boilerplate removal based on a combination of DOM-based examination, XPath expressions and rules. Given a HTML document, this library parses it, retrieves the main body text and converts it to XML or plain text, while preserving part of the text formatting and page structure. In a nutshell, with Python: ``` >>> import requests, trafilatura >>> response = requests.get('https://www.iana.org/about') >>> trafilatura.process_record(response.text) >>> # outputs main content in plain text format ... ``` On the command-line: ``` $ trafilatura -u https://www.sueddeutsche.de/politik/usa-pompeo-maas-merkel-iran-nordstream-1.4434358 $ # outputs main content in plain text format ... ``` [**Contents**](#top) - [Features](#user-content-features) - [Installation](#user-content-installation) - [With Python](#user-content-with-python) - [On the command-line](#user-content-on-the-command-line) - [Additional information](#user-content-additional-information) - [Context](#user-content-context) - [Name](#user-content-name) - [Kudos to…](#user-content-kudos-to) - [Alternatives](#user-content-alternatives) - [Contact](#user-content-contact) ## Features Scrapes the main text of web pages while preserving some structure. Also known as boilerplate removal, DOM-based content extraction, main content identification, HTML text cleaning. The purpose is to find relevant and original text sections of a web page and also to remove the noise consisting of recurring elements (headers and footers, ads, links/blogroll, etc.) Because it relies on [lxml](http://lxml.de/), trafilatura is comparatively fast. It is also robust, as the additional generic [jusText algorithm](http://corpus.tools/wiki/Justext) is used as a backup solution. The result of processing can be in plain text or XML format. In the latter case, basic formatting elements are preserved such as text formatting (bold, italic, etc.) and page structure (paragraphs, titles, lists), which can be used for further processing. _Work in progress_, currently experimental features: - Separate extraction of main text and comments - Duplicate detection at paragraph level using a least recently used (LRU) cache - Language detection on the extracted content - XML output compatible with the recommendations of the Text Encoding Initiative (XML TEI) ## Installation _trafilatura_ is a Python 3 package that is available on [PyPI](https://pypi.org/) and can be installed using pip: pip install trafilatura _(Or use \`\`pip3 install trafilatura\`\` on systems where Python 2 and 3 are both globally installed and pip refers to Python 2.)_ Direct installation of the latest version over pip is possible (see [build status](https://travis-ci.org/adbar/trafilatura)): pip install git+https://github.com/adbar/trafilatura.git ## With Python Basic use The simplest way to use trafilatura is as follows: ``` >>> import requests, trafilatura >>> response = requests.get('https://www.iana.org/about') >>> result = trafilatura.process_record(response.text) >>> print(result) # newlines preserved, TXT output >>> result = trafilatura.process_record(response.text, xml_output=True) >>> print(result) # some formatting preserved in basic XML structure ``` The only required argument is the response element, the rest is optional. It is also possible to use a previously parsed tree (i.e. a lxml.html object) as input, which is then handled seamlessly. ``` >>> from lxml import html >>> mytree = html.fromstring('

Here is the main text. It has to be long enough in order to bypass the safety checks. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

') >>> trafilatura.process_record(mytree) 'Here is the main text. It has to be long enough in order to bypass the safety checks. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\n' ``` Experimental feature: the target language can also be set using 2-letter codes ([ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)), there will be no output if the detected language of the result does not match. ``` >>> result = trafilatura.process_record(response.text, url, target_language='de') ``` For further configuration see the variables in settings.py. ## On the command-line A command-line interface is included, URLs can be used directly (\-u/--URL): ``` $ trafilatura -u https://www.sueddeutsche.de/politik/usa-pompeo-maas-merkel-iran-nordstream-1.4434358 $ # outputs main content in plain text format ... $ trafilatura --xml --URL "https://de.creativecommons.org/index.php/was-ist-cc/" $ # outputs main text with basic XML structure ... ``` You can also pipe a HTML document (and response body) to the trafilatura: ``` $ wget -qO- "https://de.creativecommons.org/index.php/was-ist-cc/" | trafilatura ``` For usage instructions see trafilatura \-h: usage: trafilatura \[-h\] \[--nocomments\] \[--xml\] \[--xmltei\] \[-u URL\] \[-v\] optional arguments: \-h, \--help show this help message and exit \--nocomments Don’t output any comments \--xml XML output \--xmltei XML TEI output \-u URL, \--URL URL custom URL download \-v, \--verbose increase output verbosity ## Additional information ### Context This module is part of methods to derive metadata from web documents in order to build text corpora for computational linguistic and NLP analysis. For more information: - Barbaresi, Adrien. “[Efficient construction of metadata-enhanced web corpora](https://hal.archives-ouvertes.fr/hal-01371704v2/document)”, Proceedings of the [10th Web as Corpus Workshop (WAC-X)](https://www.sigwac.org.uk/wiki/WAC-X), 2016. ### Name _Trafilatura_: [Italian word](https://en.wiktionary.org/wiki/trafilatura) for [wire drawing](https://en.wikipedia.org/wiki/Wire_drawing). ### Kudos to… - [lxml](http://lxml.de/) - [jusText](https://github.com/miso-belica/jusText) - [cchardet](https://github.com/PyYoshi/cChardet) & [ftfy](https://github.com/LuminosoInsight/python-ftfy) ### Alternatives Most corresponding Python modules are not actively maintained, following alternatives exist: - [dragnet](https://github.com/dragnet-org/dragnet) features combined and machine-learning approaches, but requires many dependencies as well as extensive tuning - [python-readability](https://github.com/buriy/python-readability) cleans the page and preserves some markup but is mostly geared towards news texts - [html2text](https://github.com/Alir3z4/html2text) converts HTML pages to Markup language and thus keeps the structure, though it doesn’t focus on main text extraction ### Contact Pull requests are welcome. See my [contact page](http://adrien.barbaresi.eu/contact.html) for additional details. ## Download files Download the file for your platform. If you're not sure which to choose, learn more about [installing packages](https://packaging.python.org/tutorials/installing-packages/ "External link"). ### Source Distribution ### Built Distribution Filter files by name, interpreter, ABI, and platform. If you're not sure about the file name format, learn more about [wheel file names](https://packaging.python.org/en/latest/specifications/binary-distribution-format/ "External link"). Copy a direct link to the current filters File name Interpreter ABI Platform ## File details Details for the file `trafilatura-0.0.2.tar.gz`. ### File metadata - Download URL: [trafilatura-0.0.2.tar.gz](https://files.pythonhosted.org/packages/aa/f6/aceb39cdc2c220076bcf4038949f74c98c6a37b3b45f94fd551f6515ad5c/trafilatura-0.0.2.tar.gz) - Upload date: Aug 2, 2019 - Size: 1.3 MB - Tags: Source - Uploaded using Trusted Publishing? No - Uploaded via: `twine/1.12.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.8` ### File hashes Hashes for trafilatura-0.0.2.tar.gz Algorithm Hash digest SHA256 `b212da2c9aed2037493a94035c1e16c1cf1f4068d9376b0d445a1e33077f025b` MD5 `0016a8c50779e4c4ae839d2192b61e0a` BLAKE2b-256 `aaf6aceb39cdc2c220076bcf4038949f74c98c6a37b3b45f94fd551f6515ad5c` [See more details on using hashes here.](https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode "External link") ## File details Details for the file `trafilatura-0.0.2-py3-none-any.whl`. ### File metadata - Download URL: [trafilatura-0.0.2-py3-none-any.whl](https://files.pythonhosted.org/packages/bf/81/220321dd12acd76c964de4c6044aca12d3b8b2406d90afd2514d7ca9bacb/trafilatura-0.0.2-py3-none-any.whl) - Upload date: Aug 2, 2019 - Size: 21.9 kB - Tags: Python 3 - Uploaded using Trusted Publishing? No - Uploaded via: `twine/1.12.1 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.8` ### File hashes Hashes for trafilatura-0.0.2-py3-none-any.whl Algorithm Hash digest SHA256 `b52bd76ab4cbc548b79ab294e1c9b6cab9dff395a223edd65b45535fb276c693` MD5 `7504d2bf0af37dd32239c0d4663bd3e0` BLAKE2b-256 `bf81220321dd12acd76c964de4c6044aca12d3b8b2406d90afd2514d7ca9bacb` [See more details on using hashes here.](https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode "External link")