o Q`j@s8dZddlmZddlZddlmZddlmZddlmZddlm Z dd l m Z dd l m Z dd l mZdd l mZejrkdd lmZmZmZmZmZmZmZmZmZmZddlmZddlmZeeege fZ!ej"dddZ#eddZ$ Gddde%Z&Gdddej'e#Z(e&Z)e)j*Z*e)j+Z,e)jZ-e)j.Z/dS)zMachinery for walking a filesystem. *Walking* a filesystem means recursively visiting a directory and any sub-directories. It is a fairly common requirement for copying, searching etc. See :ref:`walking` for details. )unicode_literalsN) defaultdict)deque) namedtuple) make_repr)FSError)abspath)combine)normpath) AnyCallable CollectionIteratorListOptionalMutableMappingTextTupleType)FS)Info_Fr)ZboundStepzpath, dirs, filescseZdZdZ        d,fdd ZeddZed d Zed d Zed dZ ddZ d-ddZ ddZ ddZ ddZddZddZ d-ddZ  d.d d!Zd/d"d#Zd/d$d%Z  d.d&d'Z d-d(d)Z d-d*d+ZZS)0WalkeraA walker object recursively lists directories in a filesystem. Arguments: ignore_errors (bool): If `True`, any errors reading a directory will be ignored, otherwise exceptions will be raised. on_error (callable, optional): If ``ignore_errors`` is `False`, then this callable will be invoked for a path and the exception object. It should return `True` to ignore the error, or `False` to re-raise it. search (str): If ``'breadth'`` then the directory will be walked *top down*. Set to ``'depth'`` to walk *bottom up*. filter (list, optional): If supplied, this parameter should be a list of filename patterns, e.g. ``['*.py']``. Files will only be returned if the final component matches one of the patterns. exclude (list, optional): If supplied, this parameter should be a list of filename patterns, e.g. ``['~*']``. Files matching any of these patterns will be removed from the walk. filter_dirs (list, optional): A list of patterns that will be used to match directories paths. The walk will only open directories that match at least one of these patterns. exclude_dirs (list, optional): A list of patterns that will be used to filter out directories from the walk. e.g. ``['*.svn', '*.git']``. max_depth (int, optional): Maximum directory depth to walk. FNbreadthc s|dvrtd||_|r|rtdn|r|jn|j}t|s$td||_||_||_||_ ||_ ||_ ||_ t t|dS)N)rdepthz#search must be 'breadth' or 'depth'z,on_error is invalid when ignore_errors==Truezon_error must be callable) ValueError ignore_errors_ignore_errors _raise_errorscallable TypeErroron_errorsearchfilterexclude filter_dirs exclude_dirs max_depthsuperr__init__) selfrr#r$r%r&r'r(r) __class__)/usr/lib/python3/dist-packages/fs/walk.pyr+Rs$ zWalker.__init__cCdS)zDefault on_error callback.Tr/clspatherrorr/r/r0rrzWalker._ignore_errorscCr1)z%Callback to re-raise dir scan errors.Fr/r2r/r/r0r xr6zWalker._raise_errorscCs |d}|r|ddSdS)zRCalculate the 'depth' of a directory path (number of components). /rr)stripcount)r3r4_pathr/r/r0_calculate_depth~s zWalker._calculate_depthcCst|S)aBind a `Walker` instance to a given filesystem. This *binds* in instance of the Walker to a given filesystem, so that you won't need to explicitly provide the filesystem as a parameter. Arguments: fs (FS): A filesystem object. Returns: ~fs.walk.BoundWalker: a bound walker. Example: >>> from fs import open_fs >>> from fs.walk import Walker >>> home_fs = open_fs('~/') >>> walker = Walker.bind(home_fs) >>> for path in walker.files(filter=['*.py']): ... print(path) Unless you have written a customized walker class, you will be unlikely to need to call this explicitly, as filesystem objects already have a ``walk`` attribute which is a bound walker object. Example: >>> from fs import open_fs >>> home_fs = open_fs('~/') >>> for path in home_fs.walk.files(filter=['*.py']): ... print(path) ) BoundWalker)r3fsr/r/r0binds#z Walker.bindc CsNt|jj|jdf|jdf|jdf|jdf|jdf|jdf|j df|j dfd S)NFr)rr#r$r%r&r'r(r)) rr.__name__rr#r$r%r&r'r(r)r,r/r/r0__repr__szWalker.__repr__cCs*|jdkr |j|||dS|j|||dS)zGet the walk generator.r namespaces)r$ _walk_breadth _walk_depth)r,r=r4rCr/r/r0 _iter_walks zWalker._iter_walkcCsJ|jdur||j|jrdS|jdur||j|jsdS||||S)z?Check if a directory should be considered in the walk. NF)r(matchnamer'check_open_dirr,r=r4infor/r/r0_check_open_dirs zWalker._check_open_dircCr1)adCheck if a directory should be opened. Override to exclude directories from the walk. Arguments: fs (FS): A filesystem instance. path (str): Path to directory. info (Info): A resource info object for the directory. Returns: bool: `True` if the directory should be opened. Tr/rJr/r/r0rIszWalker.check_open_dircCs&|jdur ||jkr dS||||S)z0Check if a directory contents should be scanned.NF)r)check_scan_dir)r,r=r4rKrr/r/r0_check_scan_dirszWalker._check_scan_dircCr1)aCheck if a directory should be scanned. Override to omit scanning of certain directories. If a directory is omitted, it will appear in the walk but its files and sub-directories will not. Arguments: fs (FS): A filesystem instance. path (str): Path to directory. info (Info): A resource info object for the directory. Returns: bool: `True` if the directory should be scanned. Tr/rJr/r/r0rMszWalker.check_scan_dircCs.|jdur||j|jrdS||j|jS)aCheck if a filename should be included. Override to exclude files from the walk. Arguments: fs (FS): A filesystem instance. info (Info): A resource info object. Returns: bool: `True` if the file should be included. NF)r&rGrHr%)r,r=rKr/r/r0 check_fileszWalker.check_filec csXz|j||dD]}|Vq WdSty+}z|||s WYd}~dSd}~ww)aGet an iterator of `Info` objects for a directory path. Arguments: fs (FS): A filesystem instance. dir_path (str): A path to a directory on the filesystem. namespaces (list): A list of additional namespaces to include in the `Info` objects. Returns: ~collections.Iterator: iterator of `Info` objects for resources within the given path. rBN)scandirrr#)r,r=dir_pathrCrKr5r/r/r0_scans z Walker._scanr7c cstt|}tt}|j|||d}|D]/\}}|dur=g} g} ||D] } | jr,| n| | q%t|| | V||=q|||qdS)a\Walk the directory structure of a filesystem. Arguments: fs (FS): A filesystem instance. path (str): A path to a directory on the filesystem. namespaces (list, optional): A list of additional namespaces to add to the `Info` objects. Returns: collections.Iterator: an iterator of `~fs.walk.Step` instances. The return value is an iterator of ``(, , )`` named tuples, where ```` is an absolute path to a directory, and ```` and ```` are a list of `~fs.info.Info` objects for directories and files in ````. Example: >>> home_fs = open_fs('~/') >>> walker = Walker(filter=['*.py']) >>> namespaces = ['details'] >>> for path, dirs, files in walker.walk(home_fs, namespaces) ... print("[{}]".format(path)) ... print("{} directories".format(len(dirs))) ... total = sum(info.size for info in files) ... print("{} bytes {}".format(total)) rBN)r r rlistrFis_dirappendr) r,r=r4rCr:Zdir_info_walkrQrKdirsfiles_infor/r/r0walk*s "  z Walker.walkccs>t}|j||dD]\}}|dur|js|||jVq dS)aDWalk a filesystem, yielding absolute paths to files. Arguments: fs (FS): A filesystem instance. path (str): A path to a directory on the filesystem. Yields: str: absolute path to files on the filesystem found recursively within the given directory. r4Nr rFrTrHr,r=r4_combiner:rKr/r/r0rXZ z Walker.filesccs>t}|j||dD]\}}|dur|jr|||jVq dS)aPWalk a filesystem, yielding absolute paths to directories. Arguments: fs (FS): A filesystem instance. path (str): A path to a directory on the filesystem. Yields: str: absolute path to directories on the filesystem found recursively within the given directory. r[Nr\r]r/r/r0rWlr_z Walker.dirsccsBt}|j|||d}|D]\}}|dur|||j|fVq dS)aWalk a filesystem, yielding tuples of ``(, )``. Arguments: fs (FS): A filesystem instance. path (str): A path to a directory on the filesystem. namespaces (list, optional): A list of additional namespaces to add to the `Info` objects. Yields: (str, Info): a tuple of ``(, )``. r4rCN)r rFrH)r,r=r4rCr^rVr:rKr/r/r0rK~s z Walker.infoccst|g}|j}|j}t}|j}|j} |j} |j} |j} | |} |ri|}||||dD]2}|j rU| || d}| |||rT||fV| ||||rT||||j q-| ||r_||fVq-|dfV|s#dSdS)z3Walk files using a *breadth first* search. rBrN) r appendleftpopr rRr;rLrNrOrTrH)r,r=r4rCqueuepushrbr^rRr;rLrN _check_filerrQrK_depthr/r/r0rDs4       zWalker._walk_breadthccst}|j}|j}|j}|j}|j} ||} |||||ddfg} | j} | r| d\} }}t|d}|durF|dur=|V| dfV| d=n=|jry|| | d}||| |rx||| ||rs|| |j }| |||||d| |ffn| |fVn | ||r| |fV| s&dSdS)z1Walk files using a *depth first* search. rBNr) r rRr;rLrNrOrUnextrTrH)r,r=r4rCr^rRr;rLrNrerstackrdrQZ iter_filesparentrKrfr:r/r/r0rEsF          zWalker._walk_depth)FNrNNNNNNr7Nr7)r? __module__ __qualname____doc__r+ classmethodrr r;r>rArFrLrIrNrMrOrRrZrXrWrKrDrE __classcell__r/r/r-r0r4sR    $     0   (rc@s`eZdZdZefddZddZddZ  dd d ZeZ dd d Z dddZ  dddZ d S)r<aA class that binds a `Walker` instance to a `FS` instance. Arguments: fs (FS): A filesystem instance. walker_class (type): A `~fs.walk.WalkerBase` sub-class. The default uses `~fs.walk.Walker`. You will typically not need to create instances of this class explicitly. Filesystems have a `~FS.walk` property which returns a `BoundWalker` object. Example: >>> import fs >>> home_fs = fs.open_fs('~/') >>> home_fs.walk BoundWalker(OSFS('/Users/will', encoding='utf-8')) A `BoundWalker` is callable. Calling it is an alias for `~fs.walk.BoundWalker.walk`. cCs||_||_dSrk)r= walker_class)r,r=rsr/r/r0r+s zBoundWalker.__init__cCs d|jS)NzBoundWalker({!r}))formatr=r@r/r/r0rA s zBoundWalker.__repr__cOs|j|i|}|S)z"Create a walker instance. )rs)r,argskwargswalkerr/r/r0 _make_walkerszBoundWalker._make_walkerr7NcK"|jdi|}|j|j||dS)a7 Walk the directory structure of a filesystem. Arguments: path (str): namespaces (list, optional): A list of namespaces to include in the resource information, e.g. ``['basic', 'access']`` (defaults to ``['basic']``). Keyword Arguments: ignore_errors (bool): If `True`, any errors reading a directory will be ignored, otherwise exceptions will be raised. on_error (callable): If ``ignore_errors`` is `False`, then this callable will be invoked with a path and the exception object. It should return `True` to ignore the error, or `False` to re-raise it. search (str): If ``'breadth'`` then the directory will be walked *top down*. Set to ``'depth'`` to walk *bottom up*. filter (list): If supplied, this parameter should be a list of file name patterns, e.g. ``['*.py']``. Files will only be returned if the final component matches one of the patterns. exclude (list, optional): If supplied, this parameter should be a list of filename patterns, e.g. ``['~*', '.*']``. Files matching any of these patterns will be removed from the walk. filter_dirs (list, optional): A list of patterns that will be used to match directories paths. The walk will only open directories that match at least one of these patterns. exclude_dirs (list): A list of patterns that will be used to filter out directories from the walk, e.g. ``['*.svn', '*.git']``. max_depth (int, optional): Maximum directory depth to walk. Returns: ~collections.Iterator: an iterator of ``(, , )`` named tuples, where ```` is an absolute path to a directory, and ```` and ```` are a list of `~fs.info.Info` objects for directories and files in ````. Example: >>> home_fs = open_fs('~/') >>> walker = Walker(filter=['*.py']) >>> for path, dirs, files in walker.walk(home_fs, namespaces=['details']): ... print("[{}]".format(path)) ... print("{} directories".format(len(dirs))) ... total = sum(info.size for info in files) ... print("{} bytes {}".format(total)) This method invokes `Walker.walk` with bound `FS` object. r`Nr/)rxrZr=r,r4rCrvrwr/r/r0rZs:zBoundWalker.walkcK |jdi|}|j|j|dS)aAWalk a filesystem, yielding absolute paths to files. Arguments: path (str): A path to a directory. Keyword Arguments: ignore_errors (bool): If `True`, any errors reading a directory will be ignored, otherwise exceptions will be raised. on_error (callable): If ``ignore_errors`` is `False`, then this callable will be invoked with a path and the exception object. It should return `True` to ignore the error, or `False` to re-raise it. search (str): If ``'breadth'`` then the directory will be walked *top down*. Set to ``'depth'`` to walk *bottom up*. filter (list): If supplied, this parameter should be a list of file name patterns, e.g. ``['*.py']``. Files will only be returned if the final component matches one of the patterns. exclude (list, optional): If supplied, this parameter should be a list of filename patterns, e.g. ``['~*', '.*']``. Files matching any of these patterns will be removed from the walk. filter_dirs (list, optional): A list of patterns that will be used to match directories paths. The walk will only open directories that match at least one of these patterns. exclude_dirs (list): A list of patterns that will be used to filter out directories from the walk, e.g. ``['*.svn', '*.git']``. max_depth (int, optional): Maximum directory depth to walk. Returns: ~collections.Iterator: An iterator over file paths (absolute from the filesystem root). This method invokes `Walker.files` with the bound `FS` object. r[Nr/)rxrXr=r,r4rvrwr/r/r0rXUs'zBoundWalker.filescKr{)auWalk a filesystem, yielding absolute paths to directories. Arguments: path (str): A path to a directory. Keyword Arguments: ignore_errors (bool): If `True`, any errors reading a directory will be ignored, otherwise exceptions will be raised. on_error (callable): If ``ignore_errors`` is `False`, then this callable will be invoked with a path and the exception object. It should return `True` to ignore the error, or `False` to re-raise it. search (str): If ``'breadth'`` then the directory will be walked *top down*. Set to ``'depth'`` to walk *bottom up*. filter_dirs (list, optional): A list of patterns that will be used to match directories paths. The walk will only open directories that match at least one of these patterns. exclude_dirs (list): A list of patterns that will be used to filter out directories from the walk, e.g. ``['*.svn', '*.git']``. max_depth (int, optional): Maximum directory depth to walk. Returns: ~collections.Iterator: an iterator over directory paths (absolute from the filesystem root). This method invokes `Walker.dirs` with the bound `FS` object. r[Nr/)rxrWr=r|r/r/r0rWs zBoundWalker.dirscKry)a Walk a filesystem, yielding path and `Info` of resources. Arguments: path (str): A path to a directory. namespaces (list, optional): A list of namespaces to include in the resource information, e.g. ``['basic', 'access']`` (defaults to ``['basic']``). Keyword Arguments: ignore_errors (bool): If `True`, any errors reading a directory will be ignored, otherwise exceptions will be raised. on_error (callable): If ``ignore_errors`` is `False`, then this callable will be invoked with a path and the exception object. It should return `True` to ignore the error, or `False` to re-raise it. search (str): If ``'breadth'`` then the directory will be walked *top down*. Set to ``'depth'`` to walk *bottom up*. filter (list): If supplied, this parameter should be a list of file name patterns, e.g. ``['*.py']``. Files will only be returned if the final component matches one of the patterns. exclude (list, optional): If supplied, this parameter should be a list of filename patterns, e.g. ``['~*', '.*']``. Files matching any of these patterns will be removed from the walk. filter_dirs (list, optional): A list of patterns that will be used to match directories paths. The walk will only open directories that match at least one of these patterns. exclude_dirs (list): A list of patterns that will be used to filter out directories from the walk, e.g. ``['*.svn', '*.git']``. max_depth (int, optional): Maximum directory depth to walk. Returns: ~collections.Iterable: an iterable yielding tuples of ``(, )``. This method invokes `Walker.info` with the bound `FS` object. r`Nr/)rxrKr=rzr/r/r0rKs/zBoundWalker.inforlrm) r?rnrorprr+rArxrZ__call__rXrWrKr/r/r/r0r<s   =  *%r<)0rpZ __future__rtyping collectionsrrrZ_reprrerrorsrr4r r r Z TYPE_CHECKINGr r rrrrrrrrbaserrKr ExceptionboolZOnErrorTypeVarrrobjectrZGenericr<Zdefault_walkerrZrXZ walk_filesZ walk_inforW walk_dirsr/r/r/r0s<         0  >k