Skip to content

FileStream

iotoolz.FileStream is a stream interface implemented with python's built-in open function.

iotoolz.file.FileStream

FileStream is the stream interface to the local file system with python's "open" method.

supported_schemas: Set[str]

Methods

__init__(self, uri, mode='r', buffering=-1, encoding=None, newline=None, content_type='', inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs) special

exists(self)

Whether the stream points to an existing resource.

is_dir(self)

Whether stream points to a existing dir.

is_file(self)

Whether stream points to a existing file.

iter_dir_(self)

If the current stream is a directory, this method should yield StreamInfo in the directory. Otherwise, it should yield other StreamInfo in the same directory (or level) as the current stream.

mkdir(self, mode=511, parents=False, exist_ok=False)

Create a new directory at this given path. If mode is given, it is combined with the process’ umask value to determine the file mode and access flags. If the path already exists, FileExistsError is raised.

If parents is true, any missing parents of this path are created as needed; they are created with the default permissions without taking mode into account (mimicking the POSIX mkdir -p command).

If parents is false (the default), a missing parent raises FileNotFoundError.

If exist_ok is false (the default), FileExistsError is raised if the target directory already exists.

If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file.

Parameters:

Name Type Description Default
mode int

mask mode. Defaults to 0o777.

511
parents bool

If true, creates any parents if required. Defaults to False.

False
exist_ok bool

If true, will not raise exception if dir already exists. Defaults to False.

False

read_to_iterable_(self, uri, chunk_size, fileobj, **kwargs)

read_to_iterable_ is an abstract method to implement the reading of the source resource into the a binary iterator - i.e. you will need to encode your data appropriately if it is a string.

Alternatively, 'fileobj' argument is also provided where you can write to the stream file buffer directly. In this case, you should return an empty iterable. However, this will be less efficient as the actual read will only start after all the data have been read into the buffer.

The method should return a tuple of the bytes iterator and StreamInfo object which ideally should provide the following info:

An optional dict "extras" is also provided for any other information.

Parameters:

Name Type Description Default
uri str

source uri to the resource

required
chunk_size int

size for each chunk

required
fileobj IO[bytes]

(IO[bytes]): temp fileobj for the stream

required

Exceptions:

Type Description
NotImplementedError

[description]

Returns:

Type Description
Tuple[Iterable[bytes], iotoolz._abc.StreamInfo]

Tuple[Iterable[bytes], StreamInfo]: tuple of the iterable and StreamInfo object describing the data

rmdir(self, ignore_errors=False, **kwargs)

Remove the entire directory.

stats_(self)

Retrieve the StreamInfo.

Delete and remove the resource.

write_from_fileobj_(self, uri, fileobj, size, **kwargs)

Abstract method to implement the write to the destination resource from the provided file-like object.

This file-like object only provides binary outputs - i.e. if the resources only accepts text input, you will need to decode the output inside this method.

Parameters:

Name Type Description Default
uri str

destination uri of the resource

required
fileobj IO[bytes]

file-like object to read from

required
size int

size of the data inside the file-like object

required

Exceptions:

Type Description
NotImplementedError

[description]

Returns:

Type Description
StreamInfo

StreamInfo: StreamInfo object describing the data


Last update: October 27, 2020