appyter.ext.fsspec package

Subpackages

• appyter.ext.fsspec.spec package
  • Submodules
  • appyter.ext.fsspec.spec.composable module
  • appyter.ext.fsspec.spec.mountable module
  • appyter.ext.fsspec.spec.sync_async module
  • Module contents

Submodules

appyter.ext.fsspec.chroot module

class appyter.ext.fsspec.chroot.ChrootFileSystem(*args, **kwargs)

Bases: appyter.ext.fsspec.spec.mountable.MountableAbstractFileSystem, appyter.ext.fsspec.spec.composable.ComposableAbstractFileSystem, fsspec.spec.AbstractFileSystem

chroot: update root and disallow access beyond chroot, only works on directories.

cat_file(path, start=None, end=None, **kwargs)

Get the content of a file

path: URL of file on this filesystems start, end: int Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively kwargs: passed to open().

copy(path1, path2, recursive=False, on_error=None, **kwargs)

Copy within two locations in the filesystem

on_error : “raise”, “ignore” If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

cp_file(path1, path2, **kwargs)

exists(path, **kwargs)

Is there a file at the given path

get_drs(path, **kwargs)

get_file(rpath, lpath, **kwargs)

Copy single remote file to local

info(path, **kwargs)

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation should calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

ls(path, detail=True, **kwargs)

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will

be None. * type of entry, “file”, “directory” or other

Additional information may be present, aproriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

path: str detail: bool if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str). kwargs: may have additional backend-specific options, such as version information

List of strings if detail is False, or list of directory information dicts if detail is True.

makedirs(path, exist_ok=False)

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

path: str leaf directory name exist_ok: bool (False) If False, will error if the target already exists

mkdir(path, **kwargs)

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

path: str location create_parents: bool if True, this is equivalent to makedirs kwargs: may be permissions, etc.

mount(path='', mount_dir=None, fuse=True, passthrough=True, **kwargs)

mount fallback: defer to chroot-filesystem

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)

Move file(s) from one location to another

protocol = 'chroot'

put_file(lpath, rpath, **kwargs)

Copy single file to remote

rm(path, recursive=False, maxdepth=None)

Delete files.

path: str or list of str File(s) to delete. recursive: bool If file(s) are directories, recursively delete contents and then also remove the directory maxdepth: int or None Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)

Delete a file

rmdir(path)

Remove a directory, if empty

root_marker = ''

appyter.ext.fsspec.core module

appyter.ext.fsspec.core.url_to_chroot_fs(url, pathmap=None, cached=False, appyter=None, **kwargs)

Like url_to_fs but supporting our extensions, namely: chroot filesystem path is treated as the root pathmap overlay other fsspec-compatible paths appyter create a pathmap from an appyter ipynb cached cache read/writes

appyter.ext.fsspec.core.url_to_fs_ex(url, **kwargs)

Like url_to_fs but get opts from fragment_qs

appyter.ext.fsspec.drs module

class appyter.ext.fsspec.drs.DRSFileSystem(*args, **kwargs)

Bases: appyter.ext.fsspec.spec.sync_async.SyncAsyncFileSystem, fsspec.asyn.AsyncFileSystem

protocol = 'drs'

appyter.ext.fsspec.fuse module

appyter.ext.fsspec.fuse.fs_mount(fs, fs_path='', mount_dir=None)

appyter.ext.fsspec.mapperfs module

class appyter.ext.fsspec.mapperfs.MapperFileSystem(*args, **kwargs)

Bases: appyter.ext.fsspec.spec.mountable.MountableAbstractFileSystem, fsspec.spec.AbstractFileSystem

MapperFS is the inverse of a fsspec.mapper – it lets you use a mapping to define a virtual filesystem.

cat_file(path, start=None, end=None, **kwargs)

Get the content of a file

path: URL of file on this filesystems start, end: int Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively kwargs: passed to open().

copy(path1, path2, recursive=False, on_error=None, maxdepth=None, **kwargs)

Copy within two locations in the filesystem

on_error : “raise”, “ignore” If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

exists(path, **kwargs)

Is there a file at the given path

static from_json(blob)

Recreate a filesystem instance from JSON representation

See .to_json() for the expected structure of the input

blob: str

file system instance, not necessarily of this particular class.

get_drs(path, **kwargs)

info(path, **kwargs)

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation should calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

ls(path, detail=False, **kwargs)

Aggregate results based on pathmap listing & underlying fs

makedirs(path, exist_ok=False)

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

path: str leaf directory name exist_ok: bool (False) If False, will error if the target already exists

mkdir(path, **kwargs)

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

path: str location create_parents: bool if True, this is equivalent to makedirs kwargs: may be permissions, etc.

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)

Move file(s) from one location to another

protocol = 'mapperfs'

rm(path, recursive=False, maxdepth=None)

Delete files.

path: str or list of str File(s) to delete. recursive: bool If file(s) are directories, recursively delete contents and then also remove the directory maxdepth: int or None Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)

Delete a file

rmdir(path)

Remove a directory, if empty

root_marker = ''

to_json()

JSON representation of this filesystem instance

str: JSON structure with keys cls (the python location of this class), protocol (text name of this class’s protocol, first one in case of multiple), args (positional args, usually empty), and all other kwargs as their own keys.

appyter.ext.fsspec.overlayfs module

class appyter.ext.fsspec.overlayfs.OverlayFileSystem(*args, **kwargs)

Bases: appyter.ext.fsspec.spec.mountable.MountableAbstractFileSystem, appyter.ext.fsspec.spec.composable.ComposableAbstractFileSystem, fsspec.spec.AbstractFileSystem

OverlayFS implemented with fsspec

cat_file(path, start=None, end=None, **kwargs)

Get the content of a file

path: URL of file on this filesystems start, end: int Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively kwargs: passed to open().

copy(path1, path2, recursive=False, on_error=None, maxdepth=None, **kwargs)

Copy within two locations in the filesystem

on_error : “raise”, “ignore” If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

cp_file(path1, path2, **kwargs)

exists(path, **kwargs)

Is there a file at the given path

get_drs(path, **kwargs)

get_file(rpath, lpath, **kwargs)

Copy single remote file to local

info(path, **kwargs)

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation should calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

ls(path, detail=False, **kwargs)

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will

be None. * type of entry, “file”, “directory” or other

Additional information may be present, aproriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

path: str detail: bool if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str). kwargs: may have additional backend-specific options, such as version information

List of strings if detail is False, or list of directory information dicts if detail is True.

makedirs(path, exist_ok=False)

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

path: str leaf directory name exist_ok: bool (False) If False, will error if the target already exists

mkdir(path, **kwargs)

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

path: str location create_parents: bool if True, this is equivalent to makedirs kwargs: may be permissions, etc.

mount(path='', mount_dir=None, fuse=True, **kwargs)

mount fallback: defer to chroot-filesystem

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)

Move file(s) from one location to another

protocol = 'overlayfs'

put_file(lpath, rpath, **kwargs)

Copy single file to remote

rm(path, recursive=False, maxdepth=None)

Delete files.

path: str or list of str File(s) to delete. recursive: bool If file(s) are directories, recursively delete contents and then also remove the directory maxdepth: int or None Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)

Delete a file

rmdir(path)

Remove a directory, if empty

root_marker = ''

appyter.ext.fsspec.s3fs module

appyter.ext.fsspec.sbfs module

class appyter.ext.fsspec.sbfs.SBFSBufferedFile(fs, path, mode='rb', **kwargs)

Bases: fsspec.spec.AbstractBufferedFile

class appyter.ext.fsspec.sbfs.SBFSFileSystem(*args, **kwargs)

Bases: appyter.ext.fsspec.spec.mountable.MountableAbstractFileSystem, appyter.ext.fsspec.spec.sync_async.SyncAsyncFileSystem, fsspec.asyn.AsyncFileSystem

CHUNK_SIZE = 8192

get_drs(path, **kwargs)

appyter.ext.fsspec.singleton module

class appyter.ext.fsspec.singleton.SingletonFileSystem(*args, **kwargs)

Bases: appyter.ext.fsspec.chroot.ChrootFileSystem

Instantiation of a singleton filesystem will register it by protocol after which the protocol can be used to re-instantiate the singleton instance. Serialization works for this enabling the singleton to re-register itself across process boundaries.

root_marker = ''

appyter.ext.fsspec.storage module

appyter.ext.fsspec.storage.ensure_storage(storage_uri)

appyter.ext.fsspec.test_chroot module

appyter.ext.fsspec.test_drs module

appyter.ext.fsspec.test_fuse module

appyter.ext.fsspec.test_fuseless_mount module

appyter.ext.fsspec.test_mapperfs module

appyter.ext.fsspec.test_overlayfs module

appyter.ext.fsspec.test_pathmap module

appyter.ext.fsspec.test_sbfs module

appyter.ext.fsspec.util module

async appyter.ext.fsspec.util.fsspec_cp(left_fs, left_fo, right_fs, right_fo)

async appyter.ext.fsspec.util.fsspec_read_and_run(path, op)

appyter.ext.fsspec.writecache module

class appyter.ext.fsspec.writecache.LocalTempFile(fs, path, fn=None, mode='wb', autocommit=True, seek=0, details=None)

Bases: object

A temporary local file, which will be uploaded on commit

cleanup()

close()

commit()

discard()

class appyter.ext.fsspec.writecache.WriteCacheFileSystem(*args, **kwargs)

Bases: appyter.ext.fsspec.spec.mountable.MountableAbstractFileSystem, appyter.ext.fsspec.spec.composable.ComposableAbstractFileSystem, fsspec.spec.AbstractFileSystem

cat_file(path, start=None, end=None, **kwargs)

Get the content of a file

path: URL of file on this filesystems start, end: int Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively kwargs: passed to open().

copy(path1, path2, recursive=False, on_error=None, **kwargs)

Copy within two locations in the filesystem

on_error : “raise”, “ignore” If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

cp_file(path1, path2, **kwargs)

exists(path, **kwargs)

Is there a file at the given path

get_drs(path, **kwargs)

get_file(rpath, lpath, **kwargs)

Copy single remote file to local

info(path, **kwargs)

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation should calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

ls(path, detail=False, **kwargs)

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

• full path to the entry (without protocol)

• size of the entry, in bytes. If the value cannot be determined, will

be None. * type of entry, “file”, “directory” or other

Additional information may be present, aproriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

path: str detail: bool if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str). kwargs: may have additional backend-specific options, such as version information

List of strings if detail is False, or list of directory information dicts if detail is True.

makedirs(path, exist_ok=False)

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

path: str leaf directory name exist_ok: bool (False) If False, will error if the target already exists

mkdir(path, **kwargs)

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

path: str location create_parents: bool if True, this is equivalent to makedirs kwargs: may be permissions, etc.

mount(path='', mount_dir=None, fuse=True, **kwargs)

mount fallback: defer to chroot-filesystem

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)

Move file(s) from one location to another

protocol = 'writecache'

put_file(lpath, rpath, **kwargs)

Copy single file to remote

rm(path, recursive=False, maxdepth=None)

Delete files.

path: str or list of str File(s) to delete. recursive: bool If file(s) are directories, recursively delete contents and then also remove the directory maxdepth: int or None Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)

Delete a file

rmdir(path)

Remove a directory, if empty

Module contents