appyter.ext.fsspec package

Subpackages

Submodules

appyter.ext.fsspec.chroot module

class appyter.ext.fsspec.chroot.ChrootFileSystem(*args, **kwargs)[source]

Bases: MountableAbstractFileSystem, ComposableAbstractFileSystem, AbstractFileSystem

chroot: update root and disallow access beyond chroot, only works on directories.

cat_file(path, start=None, end=None, **kwargs)[source]

Get the content of a file

Parameters

path: URL of file on this filesystems start, end: int

Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

kwargs: passed to open().

copy(path1, path2, recursive=False, on_error=None, **kwargs)[source]

Copy within two locations in the filesystem

on_error“raise”, “ignore”

If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

cp_file(path1, path2, **kwargs)[source]
exists(path, **kwargs)[source]

Is there a file at the given path

get_drs(path, **kwargs)[source]
get_file(rpath, lpath, **kwargs)[source]

Copy single remote file to local

info(path, **kwargs)[source]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns

dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

ls(path, detail=True, **kwargs)[source]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

  • full path to the entry (without protocol)

  • size of the entry, in bytes. If the value cannot be determined, will be None.

  • type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters

path: str detail: bool

if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

kwargs: may have additional backend-specific options, such as version

information

Returns

List of strings if detail is False, or list of directory information dicts if detail is True.

makedirs(path, exist_ok=False)[source]

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

Parameters

path: str

leaf directory name

exist_ok: bool (False)

If False, will error if the target already exists

mkdir(path, **kwargs)[source]

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

Parameters

path: str

location

create_parents: bool

if True, this is equivalent to makedirs

kwargs:

may be permissions, etc.

mount(path='', mount_dir=None, fuse=True, passthrough=True, **kwargs)[source]

mount fallback: defer to chroot-filesystem

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)[source]

Move file(s) from one location to another

protocol: ClassVar[str | tuple[str, ...]] = 'chroot'
put_file(lpath, rpath, **kwargs)[source]

Copy single file to remote

rm(path, recursive=False, maxdepth=None)[source]

Delete files.

Parameters

path: str or list of str

File(s) to delete.

recursive: bool

If file(s) are directories, recursively delete contents and then also remove the directory

maxdepth: int or None

Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)[source]

Delete a file

rmdir(path)[source]

Remove a directory, if empty

root_marker = ''

appyter.ext.fsspec.core module

appyter.ext.fsspec.core.url_to_chroot_fs(url, pathmap=None, cached=False, appyter=None, **kwargs)[source]

Like url_to_fs but supporting our extensions, namely: chroot filesystem path is treated as the root pathmap overlay other fsspec-compatible paths appyter create a pathmap from an appyter ipynb cached cache read/writes

appyter.ext.fsspec.core.url_to_fs_ex(url, **kwargs)[source]

Like url_to_fs but get opts from fragment_qs

appyter.ext.fsspec.drs module

class appyter.ext.fsspec.drs.DRSFileSystem(*args, **kwargs)[source]

Bases: SyncAsyncFileSystem, AsyncFileSystem

protocol: ClassVar[str | tuple[str, ...]] = 'drs'

appyter.ext.fsspec.fuse module

appyter.ext.fsspec.fuse.fs_mount(fs, fs_path='', mount_dir=None)[source]

appyter.ext.fsspec.mapperfs module

class appyter.ext.fsspec.mapperfs.MapperFileSystem(*args, **kwargs)[source]

Bases: MountableAbstractFileSystem, AbstractFileSystem

MapperFS is the inverse of a fsspec.mapper – it lets you use a mapping to define a virtual filesystem.

cat_file(path, start=None, end=None, **kwargs)[source]

Get the content of a file

Parameters

path: URL of file on this filesystems start, end: int

Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

kwargs: passed to open().

copy(path1, path2, recursive=False, on_error=None, maxdepth=None, **kwargs)[source]

Copy within two locations in the filesystem

on_error“raise”, “ignore”

If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

exists(path, **kwargs)[source]

Is there a file at the given path

static from_json(blob)[source]

Recreate a filesystem instance from JSON representation.

See .to_json() for the expected structure of the input.

Parameters

blob: str

Returns

file system instance, not necessarily of this particular class.

Warnings

This can import arbitrary modules (as determined by the cls key). Make sure you haven’t installed any modules that may execute malicious code at import time.

get_drs(path, **kwargs)[source]
info(path, **kwargs)[source]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns

dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

ls(path, detail=False, **kwargs)[source]

Aggregate results based on pathmap listing & underlying fs

makedirs(path, exist_ok=False)[source]

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

Parameters

path: str

leaf directory name

exist_ok: bool (False)

If False, will error if the target already exists

mkdir(path, **kwargs)[source]

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

Parameters

path: str

location

create_parents: bool

if True, this is equivalent to makedirs

kwargs:

may be permissions, etc.

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)[source]

Move file(s) from one location to another

protocol: ClassVar[str | tuple[str, ...]] = 'mapperfs'
rm(path, recursive=False, maxdepth=None)[source]

Delete files.

Parameters

path: str or list of str

File(s) to delete.

recursive: bool

If file(s) are directories, recursively delete contents and then also remove the directory

maxdepth: int or None

Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)[source]

Delete a file

rmdir(path)[source]

Remove a directory, if empty

root_marker = ''
to_json()[source]

JSON representation of this filesystem instance.

Parameters

include_password: bool, default True

Whether to include the password (if any) in the output.

Returns

JSON string with keys cls (the python location of this class), protocol (text name of this class’s protocol, first one in case of multiple), args (positional args, usually empty), and all other keyword arguments as their own keys.

Warnings

Serialized filesystems may contain sensitive information which have been passed to the constructor, such as passwords and tokens. Make sure you store and send them in a secure environment!

appyter.ext.fsspec.overlayfs module

class appyter.ext.fsspec.overlayfs.OverlayFileSystem(*args, **kwargs)[source]

Bases: MountableAbstractFileSystem, ComposableAbstractFileSystem, AbstractFileSystem

OverlayFS implemented with fsspec

cat_file(path, start=None, end=None, **kwargs)[source]

Get the content of a file

Parameters

path: URL of file on this filesystems start, end: int

Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

kwargs: passed to open().

copy(path1, path2, recursive=False, on_error=None, maxdepth=None, **kwargs)[source]

Copy within two locations in the filesystem

on_error“raise”, “ignore”

If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

cp_file(path1, path2, **kwargs)[source]
exists(path, **kwargs)[source]

Is there a file at the given path

get_drs(path, **kwargs)[source]
get_file(rpath, lpath, **kwargs)[source]

Copy single remote file to local

info(path, **kwargs)[source]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns

dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

ls(path, detail=False, **kwargs)[source]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

  • full path to the entry (without protocol)

  • size of the entry, in bytes. If the value cannot be determined, will be None.

  • type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters

path: str detail: bool

if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

kwargs: may have additional backend-specific options, such as version

information

Returns

List of strings if detail is False, or list of directory information dicts if detail is True.

makedirs(path, exist_ok=False)[source]

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

Parameters

path: str

leaf directory name

exist_ok: bool (False)

If False, will error if the target already exists

mkdir(path, **kwargs)[source]

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

Parameters

path: str

location

create_parents: bool

if True, this is equivalent to makedirs

kwargs:

may be permissions, etc.

mount(path='', mount_dir=None, fuse=True, **kwargs)[source]

mount fallback: defer to chroot-filesystem

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)[source]

Move file(s) from one location to another

protocol: ClassVar[str | tuple[str, ...]] = 'overlayfs'
put_file(lpath, rpath, **kwargs)[source]

Copy single file to remote

rm(path, recursive=False, maxdepth=None)[source]

Delete files.

Parameters

path: str or list of str

File(s) to delete.

recursive: bool

If file(s) are directories, recursively delete contents and then also remove the directory

maxdepth: int or None

Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)[source]

Delete a file

rmdir(path)[source]

Remove a directory, if empty

root_marker = ''

appyter.ext.fsspec.s3fs module

appyter.ext.fsspec.sbfs module

class appyter.ext.fsspec.sbfs.SBFSBufferedFile(fs, path, mode='rb', **kwargs)[source]

Bases: AbstractBufferedFile

class appyter.ext.fsspec.sbfs.SBFSFileSystem(*args, **kwargs)[source]

Bases: MountableAbstractFileSystem, SyncAsyncFileSystem, AsyncFileSystem

CHUNK_SIZE = 8192
get_drs(path, **kwargs)[source]

appyter.ext.fsspec.singleton module

class appyter.ext.fsspec.singleton.SingletonFileSystem(*args, **kwargs)[source]

Bases: ChrootFileSystem

Instantiation of a singleton filesystem will register it by protocol after which the protocol can be used to re-instantiate the singleton instance. Serialization works for this enabling the singleton to re-register itself across process boundaries.

root_marker = ''

appyter.ext.fsspec.storage module

appyter.ext.fsspec.storage.ensure_storage(storage_uri)[source]

appyter.ext.fsspec.test_chroot module

appyter.ext.fsspec.test_drs module

appyter.ext.fsspec.test_fuse module

appyter.ext.fsspec.test_fuseless_mount module

appyter.ext.fsspec.test_mapperfs module

appyter.ext.fsspec.test_overlayfs module

appyter.ext.fsspec.test_pathmap module

appyter.ext.fsspec.test_sbfs module

appyter.ext.fsspec.util module

async appyter.ext.fsspec.util.fsspec_cp(left_fs, left_fo, right_fs, right_fo)[source]
async appyter.ext.fsspec.util.fsspec_read_and_run(path, op)[source]

appyter.ext.fsspec.writecache module

class appyter.ext.fsspec.writecache.LocalTempFile(fs, path, fn=None, mode='wb', autocommit=True, seek=0, details=None)[source]

Bases: object

A temporary local file, which will be uploaded on commit

cleanup()[source]
close()[source]
commit()[source]
discard()[source]
class appyter.ext.fsspec.writecache.WriteCacheFileSystem(*args, **kwargs)[source]

Bases: MountableAbstractFileSystem, ComposableAbstractFileSystem, AbstractFileSystem

cat_file(path, start=None, end=None, **kwargs)[source]

Get the content of a file

Parameters

path: URL of file on this filesystems start, end: int

Bytes limits of the read. If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively

kwargs: passed to open().

copy(path1, path2, recursive=False, on_error=None, **kwargs)[source]

Copy within two locations in the filesystem

on_error“raise”, “ignore”

If raise, any not-found exceptions will be raised; if ignore any not-found exceptions will cause the path to be skipped; defaults to raise unless recursive is true, where the default is ignore

cp_file(path1, path2, **kwargs)[source]
exists(path, **kwargs)[source]

Is there a file at the given path

get_drs(path, **kwargs)[source]
get_file(rpath, lpath, **kwargs)[source]

Copy single remote file to local

info(path, **kwargs)[source]

Give details of entry at path

Returns a single dictionary, with exactly the same information as ls would with detail=True.

The default implementation calls ls and could be overridden by a shortcut. kwargs are passed on to `ls().

Some file systems might not be able to measure the file’s size, in which case, the returned dict will include 'size': None.

Returns

dict with keys: name (full path in the FS), size (in bytes), type (file, directory, or something else) and other FS-specific keys.

ls(path, detail=False, **kwargs)[source]

List objects at path.

This should include subdirectories and files at that location. The difference between a file and a directory must be clear when details are requested.

The specific keys, or perhaps a FileInfo class, or similar, is TBD, but must be consistent across implementations. Must include:

  • full path to the entry (without protocol)

  • size of the entry, in bytes. If the value cannot be determined, will be None.

  • type of entry, “file”, “directory” or other

Additional information may be present, appropriate to the file-system, e.g., generation, checksum, etc.

May use refresh=True|False to allow use of self._ls_from_cache to check for a saved listing and avoid calling the backend. This would be common where listing may be expensive.

Parameters

path: str detail: bool

if True, gives a list of dictionaries, where each is the same as the result of info(path). If False, gives a list of paths (str).

kwargs: may have additional backend-specific options, such as version

information

Returns

List of strings if detail is False, or list of directory information dicts if detail is True.

makedirs(path, exist_ok=False)[source]

Recursively make directories

Creates directory at path and any intervening required directories. Raises exception if, for instance, the path already exists but is a file.

Parameters

path: str

leaf directory name

exist_ok: bool (False)

If False, will error if the target already exists

mkdir(path, **kwargs)[source]

Create directory entry at path

For systems that don’t have true directories, may create an for this instance only and not touch the real filesystem

Parameters

path: str

location

create_parents: bool

if True, this is equivalent to makedirs

kwargs:

may be permissions, etc.

mount(path='', mount_dir=None, fuse=True, **kwargs)[source]

mount fallback: defer to chroot-filesystem

mv(path1, path2, recursive=False, maxdepth=None, **kwargs)[source]

Move file(s) from one location to another

protocol: ClassVar[str | tuple[str, ...]] = 'writecache'
put_file(lpath, rpath, **kwargs)[source]

Copy single file to remote

rm(path, recursive=False, maxdepth=None)[source]

Delete files.

Parameters

path: str or list of str

File(s) to delete.

recursive: bool

If file(s) are directories, recursively delete contents and then also remove the directory

maxdepth: int or None

Depth to pass to walk for finding files to delete, if recursive. If None, there will be no limit and infinite recursion may be possible.

rm_file(path)[source]

Delete a file

rmdir(path)[source]

Remove a directory, if empty

Module contents