Tryag File Manager
Home
-
Turbo Force
Current Path :
/
usr
/
lib
/
python2.4
/
site-packages
/
mx
/
TextTools
/
Upload File :
New :
File
Dir
//usr/lib/python2.4/site-packages/mx/TextTools/TextTools.pyc
mò H<c @ sB d Z d k Z d k Z d k Td k l Z d k Td k Te i d „ Z d d „ Z d „ Z d d „ Z d e i d „ Z e i d „ Z d d „ Z d „ Z d e e e e e e i e e i d „ Z d e e e e e i e d „ Z e i e e i d „ Z e e e e e i d „ Z d e e e d „ Z d e e d „ Z d e e d „ Z d e e e d „ Z d „ Z d „ Z! d „ Z" d e e# e$ d „ Z% d e e& e' e( e) ƒ d „ Z* e e+ d d f e e+ d d f d e, e- e' d d ƒ d d! f e e. e/ d e0 f d" e1 e- d d d# f f Z2 e e2 d$ „ Z3 e3 Z4 e e+ d d f e e+ d d f d e, e5 e' d d ƒ d d! f e e. e/ d e0 f d" e1 e5 d d d# f f Z6 e6 d% „ Z7 e e, e8 d f d& e, e- e# d d f e e. e/ d e0 f f Z9 e& e8 d' „ Z: d k; Z; d( f d) „ ƒ YZ< d* d+ „ Z= e> d, j o e= ƒ n d S(- s< mxTextTools - A tools package for fast text processing. Copyright (c) 2000, Marc-Andre Lemburg; mailto:mal@lemburg.com Copyright (c) 2000-2001, eGenix.com Software GmbH; mailto:info@egenix.com See the documentation for further information on copyrights, or contact the author. All Rights Reserved. N( t *( s __version__c C s | | } d } d } | d \ } } } t | ƒ d j o | d } n t | ƒ d j o | d } n t | d ƒ \ } } t | } t | ƒ | j o | d j o d } nI | d j o d } n2 t | ƒ } t | ƒ d j o | d d } n d t | ƒ d | | f | | f S( s7 Returns a pp-formatted tag table entry as string i i i i i t Tablet SubTables <table>s Here/Toi s ...s# %-15.15s : %-30s : jne=%+i : je=%+is %-.15s : %sN( R R ( t tablet it et jnet jet tt ct mt lent divmodt flagst cmdt id2cmdt typet TupleTypet Nonet repr( R R R R R R R R R R R ( ( t: /usr/lib/python2.4/site-packages/mx/TextTools/TextTools.pyt format_entry s& iÿÿÿÿc C s| g } x_ t t | ƒ ƒ D]K } | | j o | i d t | | ƒ ƒ q | i d t | | ƒ ƒ q Wt i | d ƒ d S( s; Returns a pp-formatted version of the tag table as string s --> s s N( t lt rangeR R t jR t appendR t stringt join( R R R R ( ( R t format_table6 s c C s t | ƒ GHd S( s Print the tag table N( R R ( R ( ( R t print_tagtableB s i c C sõ xî | D]æ \ } } } } t | ƒ } t | ƒ d j o | d d } n t | | | !ƒ } t | ƒ d j o | d d } n | d j o' d | d G| Gd G| G| | f GHq d | d G| Gd G| G| | f GHt | | | d ƒ q Wd S( sC Print the taglist tags for text using the given indent level i s ...i< t s |s : i N( t tagst tagR t rt subtagsR t tagnameR t textt targetR t indentt print_tags( R$ R R& R% R R R" R! R# ( ( R R' H s '#c C sø xñ | D]é } t | ƒ | j oT t | ƒ } t | ƒ d j o | d d } n d | d G| Gd t | ƒ GHq | d } | d d !\ } } t | | | !ƒ } t | ƒ d j o | d d } n d | d G| Gd | | G| | f GHq Wd S( s< Print the joinlist joins using the given indent level i( s ...R s |s (len = %i)i i i N( t joinsR R t StringTypeR R$ R R& R R! ( R( R& R) R$ R R R! ( ( R t print_joinlistY s # c C s| d g t | ƒ } xb t t | ƒ ƒ D]N } | | } t | ƒ | j o | | | <q& | d | d | d !| | <q& W| S( sÓ Return a normalized joinlist. All tuples in the joinlist are turned into real strings. The resulting list is a equivalent copy of the joinlist only consisting of strings. t i i i N( R t jlistR R R t entryR R) ( R, R) R R R- ( ( R t normlistm s !c C sW h } xJ | D]B } | | } | i | ƒ o | | i | ƒ q | g | | <q W| S( N( t dR t wt indexR t has_keyR ( R R1 R R0 R/ ( ( R t _lookup_dict„ s c C s4 g } t | ƒ } | i ƒ } t | ƒ d j oB | i d t t i | i ƒ d ƒ f ƒ | i d t d f ƒ n xž | i ƒ D] \ } } | i d ƒ t | ƒ d } x. | D]& } | i d t | d d t f ƒ q° W| i d t t f ƒ d t | t | ƒ | f | | <q€ W| i d t t f ƒ t | ƒ S( s9 Creates a lookup table that matches the words in l i R+ iÿÿÿÿi N( R R3 R R/ t keysR R R t IsInR R t Skipt itemsR t groupR R0 t Wordt MatchOkt Failt Heret Ist tuple( R R8 R/ R4 R R R R0 ( ( R t word_in_list s$ ( $$c C s÷ | | ƒ | j o | | ƒ } n | } | i } | d j oD | d j o' t | ƒ d j o | | | | ƒ Sn t | ƒ } n | t | d f | t t t | ƒ d d f | t t f f } | | | | | ƒ \ } } } | p | Sn | | ƒ S( sì A fast replacement for string.replace. what can be given as string or search object. This function is a good example for the AppendTagobj-flag usage (the taglist can be used directly as joinlist). i i iÿÿÿÿN( R t whatR) t SearchObjectt sot matcht stopR t startR t string_replaceR$ t witht sWordStartR6 t AppendTagobjt Movet ToEOFR R t foundt taglistt lastR ( R$ R@ RG RE RD RA R t joinlistR RF R R) RM RN RB R RL ( ( R t replaceª s = c C s˜ | d j o t | ƒ } n t | ƒ | j o | | ƒ } n | t | d d f f } | | | | | ƒ \ } } } | p | Sn | | | | ƒ ƒ S( s Analogon to string.replace; returns a string with all occurences of what in text[start:stop] replaced by with - uses a one entry tag-table and a Boyer-Moore-Search-object - what can be a string or a BMS/FS search object - it's faster than string.replace in those cases, where the what-string gets long and/or many replacements are found; faster meaning from a few percent up to many times as fast - start and stop define the slice of text to work in - stop defaults to len(text) i i N( RD R R R$ R R@ R) t BMSRG t sFindWordR R RE RL RM RN R RO ( R$ R@ RG RE RD R RO R R) RQ RN R RL RM ( ( R t _replace2Î s c C s t | ƒ | j o | | ƒ } n | i | ƒ } | p | Sn g } d } x2 | D]* \ } } | i | | | !| ƒ | } qT W| i | | ƒ | | d ƒ S( Ni R+ ( R R@ R) t FSt findallR$ t slicesR t xt leftt rightR RG R ( R$ R@ RG R RT R) RV R RY RW RX ( ( R t _replace3ç s c C s™ t | ƒ | j o | | ƒ } n | i | ƒ } | p | Sn d g t | ƒ } x/ t t | ƒ ƒ D] } | f | | | | <qg W| | | | ƒ ƒ S( N( R R@ R) RT RU R$ RV R R t replR R RG R RO ( R$ R@ RG R RO R RT R) RV R R[ ( ( R t _replace4ù s c C sD | d j o | | | | | | ƒ ƒ Sn | | | | | ƒ ƒ Sd S( s¨ Apply multiple replacement to a text at once. replacements must be list of tuples (replacement, left, right). It is used to replace the slice text[left:right] with the string replacement. Note that the replacements do not affect one another. Indices always refer to the original text string. Replacements must not overlap. Otherwise a ValueError is raised. N( RD R R RO R$ t replacementsRE ( R$ R] RE RD R RO ( ( R t multireplace s c C sD | d j o | | ƒ i | | | ƒ Sn | | ƒ i | | ƒ Sd S( s A faster replacement for string.find(). Uses a search object for the task. Returns the position of the first occurance of what in text[start:stop]. stop defaults to len(text). Returns -1 in case no occurance was found. N( RD R RA R@ t findR$ RE ( R$ R@ RE RD RA ( ( R R_ s c C sD | d j o | | ƒ i | | | ƒ Sn | | ƒ i | | ƒ Sd S( s Find all occurances of what in text. Uses a search object for the task. Returns a list of slice tuples (l,r) marking the all occurances in text[start:stop]. stop defaults to len(text). Returns an empty list in case no occurance was found. N( RD R RA R@ RU R$ RE ( R$ R@ RE RD RA ( ( R RU . s c C s | o | | | ƒ } n | | ƒ } | o | i | | | ƒ } n | i | | ƒ } d } g } | i } x+ | D]# \ } } | | | | !ƒ | } qt W| | | ƒ | S( s” A faster replacement for string.split(). Uses a search object for the task. Returns the result of cutting the text[start:stop] string into snippets at every sep occurance in form of a list of substrings. translate is passed to the search object as translation string. XXX convert to a C function... or even better, add as method to search objects. i N( t translateRA t sepRB RD RU R$ RE t cutsR t listR RX RY ( R$ Ra RE RD R` RA RY R Rc RB Rb R RX ( ( R t split? s c C sd x] | D]U \ } } } } | t | ƒ } | | | !| | <| o t | | | d | ƒ q q Wd S( Nt .( RM t oR R! t st prefixt strt pfxR$ t dictt _tagdict( R$ Rk Rh RM R Rf Rg R! Rj ( ( R Rl ` s c G s¬ t t | f | ƒ \ } } } | p | d | f Sn h } t } xY | D]Q \ } } } } t | ƒ } | | | !| | <| o | | t | d | ƒ qJ qJ W| | | f S( sÝ Tag a text just like the function tag() and then convert its output into a dictionary where the tagobjects reference their respective strings - this function emulates the interface of tag() - in contrast to tag() this funtion *does* make copies of the found stings - returns a tuple (rc,tagdict,next) with the same meaning of rc and next as tag(); tagdict is the new dictionary - None in case rc is 0 Re N( t applyR R$ t argst rcRM t nextR R/ Rl t tagdictRf R R! Rg Ri Rj Rk ( R$ Rn R/ Rq R Rf Rp Rg Ro R! Rj RM ( ( R Rq h s c C s t | d ƒ S( sB Return a set with all characters *except* the ones in chars. i N( t sett chars( Rs ( ( R t invset€ s c C s< | d j o t | ƒ } n | | | | | ƒ } | d j S( s Return 1 iff text[start:stop] only contains whitespace characters (as defined in Constants/Sets.py), 0 otherwise. i N( RD R R R$ t setfindt nonwhitespaceRE R ( R$ RE RD Rv Ru R ( ( R t is_whitespace† s R c C s | | | | ƒ | ƒ S( sá Eliminates newline characters and compresses whitespace characters into one space. The result is a one line text string. Tim Peters will like this function called with '-' separator ;-) N( R t setsplitR$ t collapse_sett separator( R$ Rz R Rx Ry ( ( R t collapse’ s s i s t lines iþÿÿÿs empty lineiüÿÿÿc C s | | | ƒ d S( s Split text into a list of single lines. The following combinations are considered to be line-ends: ' ', ' ', ' '; they may be used in any combination. The line-end indicators are removed from the strings prior to adding them to the list. This function allows dealing with text files from Macs, PCs and Unix origins in a portable way. i N( R R$ t linesplit_table( R$ R R} ( ( R t splitlines§ s c C s t t | | ƒ d ƒ S( s Returns the number of lines in text. Line ends are treated just like for splitlines() in a portable way. i N( R R R$ t linecount_table( R$ R ( ( R t countlinesà s t wordc C s | | | ƒ S( sª Split text into a list of single words. Words are separated by whitespace. The whitespace is stripped before adding the words to the list. N( Rx R$ t whitespace_set( R$ Rx R‚ ( ( R t splitwordsÔ s t _timerc B sw t Z d Z d Z d Z e i e i d „ Z e i e i d „ Z e i e i d „ Z e i e i d „ Z d „ Z RS( s timer class with a quite obvious interface - .start() starts a fairly accurate CPU-time timer plus an absolute timer - .stop() stops the timer and returns a tuple: the CPU-time in seconds and the absolute time elapsed since .start() was called i c C s | ƒ | _ | ƒ | _ d S( N( t timet selft atimet clockt utime( R† Rˆ R… ( ( R RE ò s c C s6 | ƒ | i | _ | ƒ | i | _ | i | i f S( N( Rˆ R† R‰ R… R‡ ( R† Rˆ R… ( ( R RD ÷ s c C s- | ƒ | i | _ | ƒ | i | _ | i S( N( Rˆ R† R‰ R… R‡ ( R† Rˆ R… ( ( R t usertimeý s c C s- | ƒ | i | _ | ƒ | i | _ | i S( N( Rˆ R† R‰ R… R‡ ( R† Rˆ R… ( ( R t abstime s c C s d | i | i f S( Ns %0.2fu %0.2fa sec.( R† R‰ R‡ ( R† ( ( R t __str__ s ( t __name__t __module__t __doc__R‰ R‡ R… Rˆ RE RD RŠ R‹ RŒ ( ( ( R R„ æ s s mxTextTools/mxTextTools.cc C s@ d „ } t | ƒ i ƒ } d k } t ƒ } d t | ƒ GHd S( Nc C sú Hx£ t t | ƒ ƒ D] } | | | | j o Pq q Wd t | ƒ t | ƒ f GHt | ƒ t | ƒ j o d t | t | ƒ ƒ GHn d t | t | ƒ ƒ GHHd Sd | GH| | d | !d t | | ƒ t | | ƒ f | | d | d !GHHd S( Ns Length mismatch: orig=%i new=%is Missing chars:s Excess chars:s Mismatch at offset %i:id s <- %s != %s ->i ( R R t origR t newR ( R R‘ R ( ( R t mismatch s Es Working on a %i byte string( R’ t opent filet readR$ R R„ R R ( R” R R$ R’ R ( ( R t _bench s . ) $t __main__(? R R t typest mxTextToolst __version__t Constants.TagTablest Constants.SetsR R R R R' R) R* R. R3 R? R RQ R RO R RP R RS RT RZ R\ R^ R_ RU Rd Rl Rq Rt t nonwhitespace_setRu Rw Rx Rr t newlinet whitespaceR{ R= t AllInSett AppendMatcht EOFR< R: R6 t _linesplit_tableR~ t linesplitRI t _linecount_tableR€ R‚ t _wordsplit_tableRƒ R… R„ R– R (" R3 Rl R RP R¥ R? R{ Rq R„ R R£ R_ R– Rd Rƒ Rš R' R R R. R¦ Rw R€ R¤ R˜ Rt RU R^ RS RZ R~ R\ R… R* ( ( R t ? sN *$!! kk = ' ÿ