euporie.core.path
Responsible for loading data from urls.
Functions
|
Add implementation class to the registry |
|
|
|
Parse and resolve a path. |
Classes
|
|
alias of |
|
|
A HTTP filesystem implementation which does not raise on errors. |
|
An |
|
PurePath subclass that can make system calls. |
|
|
|
A path for untitled files, as needed for LSP servers. |
- class euporie.core.path.HTTPFileSystem(*args, **kwargs)
Bases:
HTTPFileSystem
A HTTP filesystem implementation which does not raise on errors.
- async_impl = True
- blocksize = 4194304
- cachable = True
- cat(path, recursive=False, on_error='raise', **kwargs)
Fetch (potentially multiple) paths’ contents
- Parameters:
recursive (bool) – If True, assume the path(s) are directories, and get all the contained files
on_error ("raise", "omit", "return") – If raise, an underlying exception will be raised (converted to KeyError if the type is in self.missing_exceptions); if omit, keys with exception will simply not be included in the output; if “return”, all keys are included in the output, but the value will be bytes or an exception instance.
kwargs (passed to cat_file)
- Returns:
dict of {path (contents} if there are multiple paths)
or the path has been otherwise expanded
- cat_file(path, start=None, end=None, **kwargs)
Get the content of a file
- Parameters:
path (URL of file on this filesystems)
start (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively
end (int) – Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively
kwargs (passed to
open()
.)
- cat_ranges(paths, starts, ends, max_gap=None, on_error='return', **kwargs)
Get the contents of byte ranges from one or more files
- Parameters:
paths (list) – A list of of filepaths on this filesystems
starts (int or list) – Bytes limits of the read. If using a single int, the same value will be used to read all the specified files.
ends (int or list) – Bytes limits of the read. If using a single int, the same value will be used to read all the specified files.
- checksum(path)
Unique value for current version of file
If the checksum is the same from one moment to another, the contents are guaranteed to be the same. If the checksum changes, the contents might have changed.
This should normally be overridden; default will probably capture creation/modification timestamp (which would be good) or maybe access timestamp (which would be bad)
- classmethod clear_instance_cache()
Clear the cache of filesystem instances.
Notes
Unless overridden by setting the
cachable
class attribute to False, the filesystem class stores a reference to newly created instances. This prevents Python’s normal rules around garbage collection from working, since the instances refcount will not drop to zero untilclear_instance_cache
is called.
- static close_session(loop, session)
- copy(path1, path2, recursive=False, maxdepth=None, on_error=None, **kwargs)
Copy within two locations in the filesystem
- on_error“raise”, “ignore”
If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore
- cp(path1, path2, **kwargs)
Alias of AbstractFileSystem.copy.
- cp_file(path1, path2, **kwargs)
- created(path)
Return the created timestamp of a file as a datetime.datetime
- classmethod current()
Return the most recently instantiated FileSystem
If no instance has been created, then create one with defaults
- delete(path, recursive=False, maxdepth=None)
Alias of AbstractFileSystem.rm.
- disable_throttling = False
- disk_usage(path, total=True, maxdepth=None, **kwargs)
Alias of AbstractFileSystem.du.
- download(rpath, lpath, recursive=False, **kwargs)
Alias of AbstractFileSystem.get.
- du(path, total=True, maxdepth=None, withdirs=False, **kwargs)
Space used by files and optionally directories within a path
Directory size does not include the size of its contents.
- Parameters:
- Returns:
Dict of {path (size} if total=False, or int otherwise, where numbers)
refer to bytes used.
- encode_url(url)
- end_transaction()
Finish write transaction, non-context version
- exists(path, **kwargs)
Is there a file at the given path
- expand_path(path, recursive=False, maxdepth=None, **kwargs)
Turn one or more globs or directories into a list of all matching paths to files or directories.
kwargs are passed to
glob
orfind
, which may in turn callls
- find(path, maxdepth=None, withdirs=False, detail=False, **kwargs)
List all files below path.
Like posix
find
command without conditions
- static from_dict(dct: dict[str, Any]) AbstractFileSystem
Recreate a filesystem instance from dictionary representation.
See
.to_dict()
for the expected structure of the input.- Parameters:
- Return type:
file system instance, not necessarily of this particular class.
Warning
This can import arbitrary modules (as determined by the
cls
key). Make sure you haven’t installed any modules that may execute malicious code at import time.
- static from_json(blob: str) AbstractFileSystem
Recreate a filesystem instance from JSON representation.
See
.to_json()
for the expected structure of the input.- Parameters:
blob (str)
- Return type:
file system instance, not necessarily of this particular class.
Warning
This can import arbitrary modules (as determined by the
cls
key). Make sure you haven’t installed any modules that may execute malicious code at import time.
- property fsid
Persistent filesystem id that can be used to compare filesystems across sessions.
- get(rpath, lpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, maxdepth=None, **kwargs)
Copy file(s) to local.
Copies a specific file or tree of files (if recursive=True). If lpath ends with a “/”, it will be assumed to be a directory, and target files will go within. Can submit a list of paths, which may be glob-patterns and will be expanded.
Calls get_file for each source.
- get_file(rpath, lpath, callback=<fsspec.callbacks.NoOpCallback object>, outfile=None, **kwargs)
Copy single remote file to local
- get_mapper(root='', check=False, create=False, missing_exceptions=None)
Create key/value store based on this file-system
Makes a MutableMapping interface to the FS at the given root path. See
fsspec.mapping.FSMap
for further details.
- glob(path, maxdepth=None, **kwargs)
Find files by glob-matching.
If the path ends with ‘/’, only folders are returned.
We support
"**"
,"?"
and"[..]"
. We do not support ^ for pattern negation.The maxdepth option is applied on the first ** found in the path.
kwargs are passed to
ls
.
- head(path, size=1024)
Get the first
size
bytes from file
- info(path, **kwargs)
Give details of entry at path
Returns a single dictionary, with exactly the same information as
ls
would withdetail=True
.The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to
`ls()
.Some file systems might not be able to measure the file’s size, in which case, the returned dict will include
'size': None
.- Returns:
dict with keys (name (full path in the FS), size (in bytes), type (file,)
directory, or something else) and other FS-specific keys.
- invalidate_cache(path=None)
Discard any cached directory information
- Parameters:
path (string or None) – If None, clear all listings cached else listings at or under given path.
- isdir(path)
Is this entry directory-like?
- isfile(path)
Is this entry file-like?
- lexists(path, **kwargs)
If there is a file at the given path (including broken links)
- listdir(path, detail=True, **kwargs)
Alias of AbstractFileSystem.ls.
- property loop
- ls(url, detail=True, **kwargs)
List objects at path.
This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.
The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:
full path to the entry (without protocol)
size of the entry, in bytes. If the value cannot be determined, will be
None
.type of entry, “file”, “directory” or other
Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.
May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.
- Parameters:
- Returns:
List of strings if detail is False, or list of directory information
dicts if detail is True.
- makedir(path, create_parents=True, **kwargs)
Alias of AbstractFileSystem.mkdir.
- makedirs(path, exist_ok=False)
Recursively make directories
Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.
- mirror_sync_methods = True
- mkdir(path, create_parents=True, **kwargs)
Create directory entry at path
For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem
- mkdirs(path, exist_ok=False)
Alias of AbstractFileSystem.makedirs.
- modified(path)
Return the modified timestamp of a file as a datetime.datetime
- move(path1, path2, **kwargs)
Alias of AbstractFileSystem.mv.
- mv(path1, path2, recursive=False, maxdepth=None, **kwargs)
Move file(s) from one location to another
- open(path, mode='rb', block_size=None, cache_options=None, compression=None, **kwargs)
Return a file-like object from the filesystem
The resultant instance must function correctly in a context
with
block.- Parameters:
path (str) – Target file
mode (str like 'rb', 'w') – See builtin
open()
Mode “x” (exclusive write) may be implemented by the backend. Even if it is, whether it is checked up front or on commit, and whether it is atomic is implementation-dependent.block_size (int) – Some indication of buffering - this is a value in bytes
cache_options (dict, optional) – Extra arguments to pass through to the cache.
compression (string or None) – If given, open file using compression codec. Can either be a compression name (a key in
fsspec.compression.compr
) or “infer” to guess the compression from the filename suffix.encoding (passed on to TextIOWrapper for text mode)
errors (passed on to TextIOWrapper for text mode)
newline (passed on to TextIOWrapper for text mode)
- async open_async(path, mode='rb', size=None, **kwargs)
- pipe(path, value=None, **kwargs)
Put value into path
(counterpart to
cat
)
- pipe_file(path, value, mode='overwrite', **kwargs)
Set the bytes of given file
- put(lpath, rpath, recursive=False, callback=<fsspec.callbacks.NoOpCallback object>, maxdepth=None, **kwargs)
Copy file(s) from local.
Copies a specific file or tree of files (if recursive=True). If rpath ends with a “/”, it will be assumed to be a directory, and target files will go within.
Calls put_file for each source.
- put_file(lpath, rpath, callback=<fsspec.callbacks.NoOpCallback object>, mode='overwrite', **kwargs)
Copy single file to remote
- read_block(fn, offset, length, delimiter=None)
Read a block of bytes from
Starting at
offset
of the file, readlength
bytes. Ifdelimiter
is set then we ensure that the read starts and stops at delimiter boundaries that follow the locationsoffset
andoffset + length
. Ifoffset
is zero then we start at zero. The bytestring returned WILL include the end delimiter string.If offset+length is beyond the eof, reads to eof.
- Parameters:
Examples
>>> fs.read_block('data/file.csv', 0, 13) b'Alice, 100\nBo' >>> fs.read_block('data/file.csv', 0, 13, delimiter=b'\n') b'Alice, 100\nBob, 200\n'
Use
length=None
to read to the end of the file. >>> fs.read_block(‘data/file.csv’, 0, None, delimiter=b’n’) # doctest: +SKIP b’Alice, 100nBob, 200nCharlie, 300’See also
fsspec.utils.read_block()
- read_bytes(path, start=None, end=None, **kwargs)
Alias of AbstractFileSystem.cat_file.
- read_text(path, encoding=None, errors=None, newline=None, **kwargs)
Get the contents of the file as a string.
- Parameters:
path (str) – URL of file on this filesystems
encoding (same as open.)
errors (same as open.)
newline (same as open.)
- rename(path1, path2, **kwargs)
Alias of AbstractFileSystem.mv.
- rm(path, recursive=False, maxdepth=None)
Delete files.
- rm_file(path)
Delete a file
- rmdir(path)
Remove a directory, if empty
- root_marker = ''
- sep = '/'
- async set_session()
- sign(path, expiration=100, **kwargs)
Create a signed URL representing the given path
Some implementations allow temporary URLs to be generated, as a way of delegating credentials.
- Parameters:
- Returns:
URL – The signed URL
- Return type:
:raises NotImplementedError : if method is not implemented for a filesystem:
- size(path)
Size in bytes of file
- sizes(paths)
Size in bytes of each file in a list of paths
- start_transaction()
Begin write transaction for deferring files, non-context version
- stat(path, **kwargs)
Alias of AbstractFileSystem.info.
- tail(path, size=1024)
Get the last
size
bytes from file
- to_dict(*, include_password: bool = True) dict[str, Any]
JSON-serializable dictionary representation of this filesystem instance.
- Parameters:
include_password (bool, default True) – Whether to include the password (if any) in the output.
- Returns:
Dictionary with keys
cls
(the python location of this class),protocol (text name of this class’s protocol, first one in case of
multiple),
args
(positional args, usually empty), and all otherkeyword arguments as their own keys.
Warning
Serialized filesystems may contain sensitive information which have been passed to the constructor, such as passwords and tokens. Make sure you store and send them in a secure environment!
- to_json(*, include_password: bool = True) str
JSON representation of this filesystem instance.
- Parameters:
include_password (bool, default True) – Whether to include the password (if any) in the output.
- Returns:
JSON string with keys
cls
(the python location of this class),protocol (text name of this class’s protocol, first one in case of
multiple),
args
(positional args, usually empty), and all otherkeyword arguments as their own keys.
Warning
Serialized filesystems may contain sensitive information which have been passed to the constructor, such as passwords and tokens. Make sure you store and send them in a secure environment!
- touch(path, truncate=True, **kwargs)
Create empty file, or update timestamp
- property transaction
A context within which files are committed together upon exit
Requires the file class to implement .commit() and .discard() for the normal and exception cases.
- transaction_type
alias of
Transaction
- tree(path: str = '/', recursion_limit: int = 2, max_display: int = 25, display_size: bool = False, prefix: str = '', is_last: bool = True, first: bool = True, indent_size: int = 4) str
Return a tree-like structure of the filesystem starting from the given path as a string.
- Parameters:
path (Root path to start traversal from)
recursion_limit (Maximum depth of directory traversal)
max_display (Maximum number of items to display per directory)
display_size (Whether to display file sizes)
prefix (Current line prefix for visual tree structure)
is_last (Whether current item is last in its level)
first (Whether this is the first call (displays root path))
indent_size (Number of spaces by indent)
- Returns:
str
- Return type:
A string representing the tree structure.
Example
>>> from fsspec import filesystem
>>> fs = filesystem('ftp', host='test.rebex.net', user='demo', password='password') >>> tree = fs.tree(display_size=True, recursion_limit=3, indent_size=8, max_display=10) >>> print(tree)
- ukey(url)
Unique identifier; assume HTTP files are static, unchanging
- upload(lpath, rpath, recursive=False, **kwargs)
Alias of AbstractFileSystem.put.
- walk(path, maxdepth=None, topdown=True, on_error='omit', **kwargs)
Return all files belows path
List all files, recursing into subdirectories; output is iterator-style, like
os.walk()
. For a simple list of files,find()
is available.When topdown is True, the caller can modify the dirnames list in-place (perhaps using del or slice assignment), and walk() will only recurse into the subdirectories whose names remain in dirnames; this can be used to prune the search, impose a specific order of visiting, or even to inform walk() about directories the caller creates or renames before it resumes walk() again. Modifying dirnames when topdown is False has no effect. (see os.walk)
Note that the “files” outputted will include anything that is not a directory, such as links.
- Parameters:
path (str) – Root to recurse into
maxdepth (int) – Maximum recursion depth. None means limitless, but not recommended on link-based file-systems.
topdown (bool (True)) – Whether to walk the directory tree from the top downwards or from the bottom upwards.
on_error ("omit", "raise", a callable) – if omit (default), path with exception will simply be empty; If raise, an underlying exception will be raised; if callable, it will be called with a single OSError instance as argument
kwargs (passed to
ls
)
- write_bytes(path, value, **kwargs)
Alias of AbstractFileSystem.pipe_file.
- write_text(path, value, encoding=None, errors=None, newline=None, **kwargs)
Write the text to the given file.
An existing file will be overwritten.
- class euporie.core.path.NoRaiseClientResponse(method: str, url: URL, *, writer: asyncio.Task[None] | None, continue100: asyncio.Future[bool] | None, timer: BaseTimerContext, request_info: RequestInfo, traces: List[Trace], loop: AbstractEventLoop, session: ClientSession)
Bases:
ClientResponse
An
aiohttp
client response which does not raise on >=400 status responses.- ATTRS = frozenset({'_content_dict', '_content_type', '_stored_content_type'})
- content: StreamReader = None
- content_disposition
- property cookies: SimpleCookie
- headers
- history
A sequence of of responses, if redirects occurred.
- host
- async json(*, encoding: str | None = None, loads: ~typing.Callable[[str], ~typing.Any] = <function loads>, content_type: str | None = 'application/json') Any
Read and decodes JSON response.
- links
- raw_headers
- real_url
- request_info
- async start(connection: Connection) ClientResponse
Start response processing.
- async text(encoding: str | None = None, errors: str = 'strict') str
Read response payload and decode.
- url
- url_obj
- class euporie.core.path.UntitledPath(*args, protocol: str | None = None, **storage_options: Any)
Bases:
UPath
A path for untitled files, as needed for LSP servers.
- absolute() Self
Return an absolute version of this path No normalization or symlink resolution is performed.
Use resolve() to resolve symlinks and remove ‘..’ segments.
- property anchor
The concatenation of the drive and root, or ‘’.
- as_posix()
Return the string representation of the path with forward (/) slashes.
- chmod(mode: int, *, follow_symlinks: bool = True) None
Change the permissions of the path, like os.chmod().
- property drive
The drive prefix (letter or UNC path), if any.
- expanduser() Self
Return a new path with expanded ~ and ~user constructs (as returned by os.path.expanduser)
- classmethod from_uri(uri)
Return a new path from the given ‘file’ URI.
- property fs: AbstractFileSystem
The cached fsspec filesystem instance for the path.
- full_match(pattern, *, case_sensitive=None)
Return True if this path matches the given glob-style pattern. The pattern is matched against the entire path.
- glob(pattern: str, *, case_sensitive=None) Generator[UPath, None, None]
Iterate over this subtree and yield all existing files (of any kind, including directories) matching the given relative pattern.
- hardlink_to(target: str | PathLike[str] | UPath) None
Make this path a hard link pointing to the same file as target.
Note the order of arguments (self, target) is the reverse of os.link’s.
- is_file() bool
Whether this path is a regular file (also True for symlinks pointing to regular files).
- is_relative_to(other, /, *_deprecated) bool
Return True if the path is relative to another path or False.
- is_reserved() bool
Return True if the path contains one of the special names reserved by the system, if any.
- iterdir() Generator[UPath, None, None]
Yield path objects of the directory contents.
The children are yielded in arbitrary order, and the special entries ‘.’ and ‘..’ are not included.
- joinpath(*pathsegments: str | os.PathLike[str]) Self
Combine this path with one or several arguments, and return a new path representing either a subpath (if all arguments are relative paths) or a totally different path (if one of the arguments is anchored).
- lchmod(mode: int) None
Like chmod(), except if the path points to a symlink, the symlink’s permissions are changed, rather than its target’s.
- lstat() UPathStatResult
Like stat(), except if the path points to a symlink, the symlink’s status information is returned, rather than its target’s.
- match(path_pattern, *, case_sensitive=None)
Return True if this path matches the given pattern. If the pattern is relative, matching is done from the right; otherwise, the entire path is matched. The recursive wildcard ‘**’ is not supported by this method.
- open(mode: str = 'r', *args: Any, **fsspec_kwargs: Any) IO[Any]
Open the file pointed by this path and return a file object, as the built-in open() function does.
- Parameters:
mode – Opening mode. Default is ‘r’.
buffering – Default is the block size of the underlying fsspec filesystem.
encoding – Encoding is only used in text mode. Default is None.
errors – Error handling for encoding. Only used in text mode. Default is None.
newline – Newline handling. Only used in text mode. Default is None.
**fsspec_kwargs – Additional options for the fsspec filesystem.
- property parent
The logical parent of the path.
- property parents
A sequence of this path’s logical parents.
- parser = <wrapped class AnyProtocolFileSystemFlavour>
- property parts
An object providing sequence-like access to the components in the filesystem path.
- read_bytes()
Open the file in bytes mode, read it, and close the file.
- read_text(encoding=None, errors=None, newline=None)
Open the file in text mode, read it, and close the file.
- readlink() Self
Return the path to which the symbolic link points.
- relative_to(other, /, *_deprecated, walk_up=False) Self
Return the relative path to another path identified by the passed arguments. If the operation is not possible (because this is not related to the other path), raise ValueError.
The walk_up parameter controls whether .. may be used to resolve the path.
- rename(target: str | os.PathLike[str] | UPath, *, recursive: bool = <object object>, maxdepth: int | None = <object object>, **kwargs: Any) Self
Rename this path to the target path.
The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, not the directory of the Path object.
Returns the new Path instance pointing to the target path.
- replace(target: str | PathLike[str] | UPath) UPath
Rename this path to the target path, overwriting if that path exists.
The target path may be absolute or relative. Relative paths are interpreted relative to the current working directory, not the directory of the Path object.
Returns the new Path instance pointing to the target path.
- resolve(strict: bool = False) Self
Make the path absolute, resolving all symlinks on the way and also normalizing it.
- rglob(pattern: str, *, case_sensitive=None) Generator[UPath, None, None]
Recursively yield all existing files (of any kind, including directories) matching the given relative pattern, anywhere in this subtree.
- property root
The root of the path, if any.
- samefile(other_path) bool
Return whether other_path is the same or not as this file (as returned by os.path.samefile()).
- stat(*, follow_symlinks=True) UPathStatResult
Return the result of the stat() system call on this path, like os.stat() does.
- property stem
The final path component, minus its last suffix.
- property suffix
The final component’s last suffix, if any.
This includes the leading period. For example: ‘.txt’
- property suffixes
A list of the final component’s suffixes, if any.
These include the leading periods. For example: [‘.tar’, ‘.gz’]
- symlink_to(target: str | PathLike[str] | UPath, target_is_directory: bool = False) None
Make this path a symlink pointing to the target path. Note the order of arguments (link, target) is the reverse of os.symlink.
- touch(mode=438, exist_ok=True) None
Create this file with the given access mode, if it doesn’t exist.
- unlink(missing_ok: bool = False) None
Remove this file or link. If the path is a directory, use rmdir() instead.
- walk(top_down=True, on_error=None, follow_symlinks=False)
Walk the directory tree from this directory, similar to os.walk().
- with_name(name)
Return a new path with the file name changed.
- with_segments(*pathsegments: str | os.PathLike[str]) Self
Construct a new path object from any number of path-like objects. Subclasses may override this method to customize how new path objects are created from methods like iterdir().
- with_stem(stem)
Return a new path with the stem changed.
- with_suffix(suffix)
Return a new path with the file suffix changed. If the path has no suffix, add given suffix. If the given suffix is an empty string, remove the suffix from the path.
- write_bytes(data)
Open the file in bytes mode, write to it, and close the file.
- write_text(data, encoding=None, errors=None, newline=None)
Open the file in text mode, write to it, and close the file.