Tryag File Manager
Home
-
Turbo Force
Current Path :
/
usr
/
lib
/
python2.4
/
site-packages
/
lxml
/
html
/
Upload File :
New :
File
Dir
//usr/lib/python2.4/site-packages/lxml/html/clean.pyo
m ZGc @ sc d Z d k Z d k Z d k Z d k l Z d k l Z d k l Z l Z y e Wn e j o d k l Z n Xd d d d d d d g Z e i d e i e i B Z e i d e i Z e i d e i Z e i d i Z e i d e i e i B Z e i d Z e i d Z d e f d YZ e Z e i Z e i d e i e i d e i g Z d d d d d d g Z e i d e i e i d e i e i d g Z! d g Z" e e e! e" d! Z# d" Z$ d# Z% e# i e% _ d d d g Z& d$ g Z' d% e& e' d& d' Z( d( Z) d) Z* e i d* e i Z+ d+ Z, d S(, sc A cleanup tool for HTML. Removes unwanted tags and content. See the `Cleaner` class for details. N( s etree( s defs( s fromstrings tostring( s Sett clean_htmlt cleant Cleanert autolinkt autolink_htmlt word_breakt word_break_htmls expression\s*\(.*?\)s @\s*imports: \s*(?:javascript|jscript|livescript|vbscript|about|mocha):s \s+s \[if[\s\n\r]+.*?][\s\n\r]*>s descendant-or-self::*[@style]s^ descendant-or-self::a[normalize-space(@href) and substring(normalize-space(@href),1,1) != '#']c B s: t Z d Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z f Z e d d g Z d Z e d d d d d d d g d d d d d d d d Z d Z d Z d Z d Z d Z! e d Z" d Z# e$ i% d e$ i& i' Z( d Z) d Z* RS( s Instances cleans the document of each of the possible offending elements. The cleaning is controlled by attributes; you can override attributes in a subclass, or set them in the constructor. ``scripts``: Removes any ``<script>`` tags. ``javascript``: Removes any Javascript, like an ``onclick`` attribute. ``comments``: Removes any comments. ``style``: Removes any style tags or attributes. ``links``: Removes any ``<link>`` tags ``meta``: Removes any ``<meta>`` tags ``page_structure``: Structural parts of a page: ``<head>``, ``<html>``, ``<title>``. ``processing_instructions``: Removes any processing instructions. ``embedded``: Removes any embedded objects (flash, iframes) ``frames``: Removes any frame-related tags ``forms``: Removes any form tags ``annoying_tags``: Tags that aren't *wrong*, but are annoying. ``<blink>`` and ``<marque>`` ``remove_tags``: A list of tags to remove. ``allow_tags``: A list of tags to include (default include all). ``remove_unknown_tags``: Remove any tags that aren't standard parts of HTML. ``safe_attrs_only``: If true, only include 'safe' attributes (specifically the list from `feedparser <http://feedparser.org/docs/html-sanitization.html>`_). ``add_nofollow``: If true, then any <a> tags will have ``rel="nofollow"`` added to them. ``host_whitelist``: A list or set of hosts that you can use for embedded content (for content like ``<object>``, ``<link rel="stylesheet">``, etc). You can also implement/override the method ``allow_embedded_url(el, url)`` or ``allow_element(el)`` to implement more complex rules for what can be embedded. Anything that passes this test will be shown, regardless of the value of (for instance) ``embedded``. Note that this parameter might not work as intended if you do not make the links absolute before doing the cleaning. ``whitelist_tags``: A set of tags that can be included with ``host_whitelist``. The default is ``iframe`` and ``embed``; you may wish to include other tags like ``script``, or you may want to implement ``allow_embedded_url`` for more control. Set to None to include all tags. This modifies the document *in place*. t iframet embedc K s[ xT | i D]F \ } } t | | p t d | | f n t | | | q Wd S( Ns Unknown parameter: %s=%r( t kwt itemst namet valuet hasattrt selft TypeErrort setattr( R R R R ( ( t3 /usr/lib/python2.4/site-packages/lxml/html/clean.pyt __init__ s t scriptt srct linkt hreft applett codet objectt layert ac C s t | d o | i } n x | i d D] } d | _ q0 W| i p | i | n t } t | i p f } | i o t | i } n t } | i o | i d n | i ob t t i } xP | i D]> } | i } x, | i D] } | | j o | | =q q Wq Wn | i o| i pV xS | i D]A } | i } x/ | i D]! } | i d o | | =qgqgWqKWn | i | i d t | i pNx t | D]x } | i d } t i d | } t! i d | } | i" | o | i d =q| | j o | i d | qqWx t# | i d D] } | i d d i$ i% d j o | i& qSn | i' p d } t i d | } t! i d | } | i" | o d | _' qS| | j o | | _' qSqSWqn | i p | i( o | i t) i* n | i( o | i t) i+ n | i o2 | i d x" t | D] } | i d =qpWn | i, o | i d nf | i p | i oQ xN t# | i d D]3 } d | i d d i$ j o | i& qqWn | i- o | i d n | i. o | i/ d% n | i0 o x{ t# | i d D]d } t } | i2 } x. | d$ j o | i d& j o | i2 } qyW| d$ j o | i& q^q^W| i/ d' | i/ d( n | i5 o | i/ t i6 n | i7 o | i d | i/ d) n | i8 o | i/ d* n g } g } x | i D]v } | i | j o( | i; | o q^n | i< | q^| i | j o( | i; | o q^n | i< | q^q^W| o: | d | j o) | i= d } d! | _ | i i> nS | oK | d | j o: | i= d } | i d j o d! | _ n | i> n x | D] } | i& qrWx | D] } | i? qW| i } | i@ o* | o tA d" n t t iB } n | o] g } x5 | i D]' } | i | j o | i< | qqWx | D] } | i? q*Wn | iD o? x< tE | D]* } | iF | p | i d d# qYqYWn d$ S(+ s& Cleans the document. t getroott imaget imgR t ont resolve_base_hreft stylet t types text/javascripts /* deleted */R t stylesheett relt metat headt htmlt titlet paramR R R R R t formt buttont inputt selectt textareat blinkt marquei t divsI It does not make sense to pass in both allow_tags and remove_unknown_tagst nofollowN( s heads htmls title( s applets object( s applet( R R s layers objectR* ( R, s inputR. R/ ( R0 R1 (G R t docR t itert elt tagR t commentst kill_conditional_commentst sett kill_tagst remove_tagst allow_tagst scriptst addt safe_attrs_onlyt defst safe_attrst attribt keyst anamet javascriptt startswitht rewrite_linkst _remove_javascript_linkt FalseR! t _find_styled_elementst gett oldt _css_javascript_ret subt newt _css_import_ret _has_sneaky_javascriptt listt lowert stript drop_treet textt processing_instructionst etreet Commentt ProcessingInstructiont linksR& t page_structuret updatet embeddedt found_parentt getparentt parentt Nonet framest frame_tagst formst annoying_tagst _removet _killt allow_elementt appendt popt cleart drop_tagt remove_unknown_tagst ValueErrort tagst badt add_nofollowt _find_external_linkst allow_follow( R R4 R6 RM R; R= Rh RP R< Rb RB Ri RE R` Rr RC ( ( R t __call__ s % c C s t S( sF Override to suppress rel="nofollow" on some anchors. N( RJ ( R t anchor( ( R Ru h s c C s | i | i j o t Sn | i | i } t | t t f oR xG | D]? } | i | } | p t Sn | i | | p t SqH qH Wt Sn/ | i | } | p t Sn | i | | Sd S( N( R6 R7 R t _tag_link_attrsRJ t attrt isinstanceRS t tuplet one_attrRL t urlt allow_embedded_urlt True( R R6 Ry R} R| ( ( R Rj n s c C s | i d j o | i | i j o t Sn t i | \ } } } } } | i i d d d } | d j o t Sn | | i j o t Sn t S( Nt :i i t httpt https( R R ( R t whitelist_tagsRc R6 R7 RJ t urlparset urlsplitR} t schemet netloct patht queryt fragmentRT t splitt host_whitelistR ( R R6 R} R R R R R ( ( R R~ s # c C s# g } | i | d t i d S( s IE conditional comments basically embed HTML that the parser doesn't normally see. We can't allow anything like that, so we'll kill any comments that could be conditional. c C s t i | i S( N( t _conditional_comment_ret searchR6 RW ( R6 ( ( R t <lambda> s N( Rr R t _kill_elementsR4 RY RZ ( R R4 Rr ( ( R R9 s c C s] g } x5 | i | D]$ } | | o | i | q q Wx | D] } | i qE Wd S( N( Rr R4 R5 t iterateR6 t conditionRk RV ( R R4 R R R6 Rr ( ( R R s c C s+ t d | } t i | o d Sn | S( NR" ( t _substitute_whitespaceR RP t _javascript_scheme_reR ( R R RP ( ( R RI s s /\*.*?\*/c C sm | i d | } | i d d } t d | } | i } d | j o t Sn d | j o t Sn t S( s Depending on the browser, stuff like ``e x p r e s s i o n(...)`` can get interpreted, or ``expre/* stuff */ssion(...)``. This checks for attempt to do stuff like this. Typically the response will be to kill the entire style; if you have just a bit of Javascript in the style another rule will catch that and remove only the Javascript from the style; this catches more sneaky attempts. R" s \s javascript:s expression(N( R t _substitute_commentsR! t replaceR RT R RJ ( R R! ( ( R RR s c C sb t | t o t } t | } n t } t i | } | | | o t | Sn | Sd S( N( Rz R( t basestringR t return_stringt fromstringR4 RJ t copyt deepcopyR t tostring( R R( R4 R ( ( R R s (+ t __name__t __module__t __doc__R R> RF R8 RJ R! R\ R& R] RX R_ Rd Rf Rg Rc R< R= Ro R@ Rs R R: R R t dictRx Rv Ru Rj R~ R9 R RI t ret compilet SRO R RR R ( ( ( R R C sL O sD (?P<body>https?://(?P<host>[a-z0-9._-]+)(?:/[/\-_.,a-z0-9%&?;=~]*)?)s9 mailto:(?P<body>[a-z0-9._-]+@(?P<host>[a-z0-9_._]+[a-z]))R/ t preR R' R. R s ^localhosts \bexample\.(?:com|org|net)$s ^127\.0\.0\.1$t nolinkc C s[ | i | j o d Sn | i d } | o6 | i } x' | D] } | | j o d SqA qA Wn x t | D] } t | d | d | d | d | | i o` t | i | | d | i \ } } | o1 | | _ | i | } | | | d | d +qqq qq W| i oF t | i | | d | i \ } } | o | | _ | | d *qWn d S( s Turn any URLs into links. It will search for links identified by the given regular expressions (by default mailto and http(s) links). It won't link text in an element in avoid_elements, or an element with a class in avoid_classes. It won't link to anything with a host that matches one of the regular expressions in avoid_hosts (default localhost and 127.0.0.1). If you pass in an element, the elements tail will not be substituted, only the contents of the element. Nt classt link_regexest avoid_elementst avoid_hostst avoid_classest factoryi i ( R6 R7 R RL t class_nameR R t match_classRS t childR R R t tailt _link_textt makeelementRW t tail_childrent indext pre_children( R6 R R R R R RW R R R R R ( ( R R s6 $ ! $ c C s= d } g } d } x d \ } } x | D] } | } xn | i | d | } | d j o Pn | i d } x0 | D]' } | i | o | i } Pqt qt WPq7 W| d j o q( n | d j p | i | j o | } | i } q( q( W| d j o# | o | | d _ n | } Pn | i d } | i } | i d p | i d o | d 8} | d } n | | i } | o | | d _ n | } | d } | i d | | i d } | p | } n | i d p | i d o | d } n | | _ | i | | | } q W| | f S( NR" i i t post hostit .t ,R R t body( NN( t leading_textR\ t last_posRc t best_matcht best_posR t regext regex_posR RW t matcht groupR R t host_regext endt startR R t endswitht prev_textR Rw R: R Rk ( RW R R R R R R R R R\ R R R R R R R Rw R ( ( R R sn c O sh t | t o t | } t } n t i | } t } t | | | | o t | Sn | Sd S( N( Rz R( R R R4 R R R R RJ R t argsR R ( R( R R R4 R ( ( R R I s t nobreaki( u c C s | i t j o d Sn | i d } | oN t } | i } x& | D] } | | j o t } PqG qG W| o d Sq| n | i o t | i | | | _ n xV | D]N } t | d | d | d | d | | i o t | i | | | _ q q Wd S( s Breaks any long words found in the body of the text (not attributes). Doesn't effect any of the tags in avoid_elements, by default ``<textarea>`` and ``<pre>`` Breaks words by inserting ​, which is a unicode character for Zero Width Space character. This generally takes up no space in rendering, but does copy as a space, and in monospace contexts usually takes up space. See http://www.cs.tut.fi/~jkorpela/html/nobr.html for a discussion NR t max_widthR R t break_character( R6 R7 t _avoid_word_break_elementsRL R RJ t dont_breakR R t avoidR RW t _break_textR R R R R R ( R6 R R R R R R R R ( ( R R _ s0 c O s& t | } t | | | t | S( N( R R( R4 R R R R ( R( R R R4 ( ( R R s c C s\ | i } xI | D]A } t | | j o( t | | | } | i | | } q q W| S( N( RW R t wordst wordt lenR t _insert_breakR t replacementR ( RW R R R R R ( ( R R s s [^a-z]c C s | } d } x t | | j o | | } t t i | } | o9 | d } | i | d j o | | i } q n | | | 7} | t | } q W| | 7} | S( NR" ii ( R t orig_wordt resultR t widthR RS t _break_prefer_ret finditert breakst last_breakR R ( R R R R R R R R ( ( R R s (- R R R R t lxmlRY t lxml.htmlRA R R R: t NameErrort setst Sett __all__R R t IRN RQ R RO R R t XPathRK Rt R R R R t _link_regexest _avoid_elementst _avoid_hostst _avoid_classesR R R R t _avoid_word_break_classesR R R R R (! RQ R: R R R RK R R R R R R R Rt R RA R R RY R R R R R R RN R R R R R R R ( ( R t ? sJ *6 - : )