AbcStream¶

iotoolz.AbcStream is the base abstract class for all the concrete streams.

`iotoolz._abc.AbcStream` ¶

AbcStream is an abstract class which mimics python's native open function very closely.

Attributes¶

`closed: bool` `property` `readonly` ¶

True if the stream is closed.

`content_type: str` `property` `readonly` ¶

Resource media type (e.g. application/json, text/plain).

Returns:

Type	Description
`str`	str: string describing the media type of the resource.

`encoding: str` `property` `readonly` ¶

Text encoding to use to decode internal binary data into text outputs.

Returns:

Type	Description
`str`	str: text encoding.

`etag: str` `property` `readonly` ¶

Identifier for a specific version of a resource.

`info: StreamInfo` `property` `readonly` ¶

Stream info like content type, encoding, and etag.

Returns:

Type	Description
`StreamInfo`	StreamInfo: StreamInfo object for the current stream.

`INMEM_SIZE: int` ¶

`size: int` `property` `readonly` ¶

`supported_schemas: Set[str]` ¶

Methods¶

`init(self, uri, mode='r', buffering=-1, encoding=None, newline=None, content_type='', inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs)` `special` ¶

Creates a new instance of AbcStream.

Parameters:

Name	Type	Description	Default
`uri`	`str`	uri string to the resource.	required
`mode`	`str`	same as "open" - supports depends on the actual implementation. Defaults to "r".	`'r'`
`buffering`	`int`	same as "open". Defaults to -1.	`-1`
`encoding`	`str`	encoding used to decode bytes to str. Defaults to None.	`None`
`newline`	`str`	same as "open". Defaults to None.	`None`
`content_type`	`str`	mime type for the resource. Defaults to "".	`''`
`inmem_size`	`int`	max size before buffer rollover from mem to disk. Defaults to None (i.e. never - may raise MemoryError).	`None`
`delimiter`	`Union[str, bytes]`	delimiter used for determining line boundaries. Defaults to None.	`None`
`chunk_size`	`int`	chunk size when iterating bytes stream. Defaults to io.DEFAULT_BUFFER_SIZE.	`8192`
`etag`	`str`	etag for the stream content. Defaults to "".	`''`
`**kwargs`		Additional keyword arguments to the stream (depends on implementation)	`{}`

`clone(self, uri=None, info=None, **kwargs)` ¶

Creates a new stream with the same init args as the current stream.

Returns:

Type	Description
`AbcStream`	AbcStream: new stream with same init args.

`close(self)` ¶

Flush and close this stream. This method has no effect if the file is already closed. Once the file is closed, any operation on the file (e.g. reading or writing) will raise a ValueError.

As a convenience, it is allowed to call this method more than once; only the first call, however, will have an effect.

`exists(self)` ¶

Whether the path points to an existing resource.

`flush(self)` ¶

Flush the write buffers of the stream if applicable. This does nothing for read-only and non-blocking streams.

`glob(self, pattern='*')` ¶

Glob the given relative pattern in the directory represented by this path, yielding all matching files (of any kind).

Parameters:

Name	Type	Description	Default
`pattern`	`str`	unix shell style pattern. Defaults to "*".	`'*'`

Returns:

Type	Description
`Iterator[AbcStream]`	Iterator["AbcStream"]: iterator of streams relative to current stream

Yields

AbcStream: stream object

`is_dir(self)` ¶

Whether stream points to a existing dir.

`is_empty(self)` ¶

True if current buffer is empty.

Returns:

Type	Description
`bool`	bool: whether current buffer is empty.

`is_file(self)` ¶

Whether stream points to a existing file.

`iter_bytes(self)` ¶

Returns an iterator which yields a bytes stream. Will load the resource into buffer if needed.

Yields

Iterator[bytes]: bytes stream from the start position

`iter_dir(self)` ¶

If the current stream is a directory, this method will yield all Stream in the directory. Otherwise, it should yield all Stream in the same directory (or level) as the current stream.

`iter_dir_(self)` ¶

If the current stream is a directory, this method should yield StreamInfo in the directory. Otherwise, it should yield other StreamInfo in the same directory (or level) as the current stream.

`mkdir(self, mode=511, parents=False, exist_ok=False)` ¶

This abstract method should mimics pathlib.mkdir method for different streams.

Create a new directory at this given path. If mode is given, it is combined with the process’ umask value to determine the file mode and access flags. If the path already exists, FileExistsError is raised.

If parents is true, any missing parents of this path are created as needed; they are created with the default permissions without taking mode into account (mimicking the POSIX mkdir -p command).

If parents is false (the default), a missing parent raises FileNotFoundError.

If exist_ok is false (the default), FileExistsError is raised if the target directory already exists.

If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file.

Parameters:

Name	Type	Description	Default
`mode`	`int`	mask mode. Defaults to 0o777.	`511`
`parents`	`bool`	If true, creates any parents if required. Defaults to False.	`False`
`exist_ok`	`bool`	If true, will not raise exception if dir already exists. Defaults to False.	`False`

Exceptions:

Type	Description
`NotImplementedError`	[description]

`open(uri, mode='r', buffering=-1, encoding=None, newline=None, inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs)` `classmethod` ¶

Creates a new instance of AbcStream. This mimics python's open method but supports additional keyword arguments.

Parameters:

Name	Type	Description	Default
`uri`	`str`	uri string to the resource.	required
`mode`	`str`	same as "open" - supports depends on the actual implementation. Defaults to "r".	`'r'`
`buffering`	`int`	same as "open". Defaults to -1.	`-1`
`encoding`	`str`	encoding used to decode bytes to str. Defaults to None.	`None`
`newline`	`str`	same as "open". Defaults to None.	`None`
`content_type`	`str`	mime type for the resource. Defaults to "".	required
`inmem_size`	`int`	max size before buffer rollover from mem to disk. Defaults to None (i.e. never - may raise MemoryError).	`None`
`delimiter`	`Union[str, bytes]`	delimiter used for determining line boundaries. Defaults to None.	`None`
`chunk_size`	`int`	chunk size when iterating bytes stream. Defaults to io.DEFAULT_BUFFER_SIZE.	`8192`
`etag`	`str`	etag for the stream content. Defaults to "".	`''`
`**kwargs`		Additional keyword arguments to the stream (depends on implementation)	`{}`

Returns:

Type	Description
`AbcStream`	AbcStream: new instance of AbcStream

`peek(self, size=None)` ¶

Return bytes from the stream without advancing the position.

Parameters:

Name	Type	Description	Default
`size`	`Optional[int]`	number of bytes to return. Return default	`None`

Returns:

Type	Description
`bytes`	bytes: data starting from current stream pos.

`pipe(self, sink, text_mode=False)` ¶

Pipes the data from the current stream object into any file-like object.

Parameters:

Name	Type	Description	Default
`sink`	`IO`	Any file-like object or AbcStream object.	required
`text_mode`	`bool`	If True, writes string to sink rather than bytes. Defaults to False.	`False`

Exceptions:

Type	Description
`ValueError`	sink for pipe must be a filelike object - i.e. has write method

Returns:

Type	Description
`IO`	IO: file-like object (i.e. sink) that is piped into.

`read(self, size=-1)` ¶

Read and return up to size bytes. If the argument is omitted, None, or negative, data is read and returned until EOF is reached. An empty bytes object is returned if the stream is already at EOF.

Parameters:

Name	Type	Description	Default
`size`	`Optional[int]`	number of bytes to read. Defaults to -1.	`-1`

Returns:

Type	Description
`Union[str, bytes, bytearray]`	Union[str, bytes, bytearray]: data starting from current stream pos.

`read_to_iterable_(self, uri, chunk_size, fileobj, **kwargs)` ¶

read_to_iterable_ is an abstract method to implement the reading of the source resource into the a binary iterator - i.e. you will need to encode your data appropriately if it is a string.

Alternatively, 'fileobj' argument is also provided where you can write to the stream file buffer directly. In this case, you should return an empty iterable. However, this will be less efficient as the actual read will only start after all the data have been read into the buffer.

The method should return a tuple of the bytes iterator and StreamInfo object which ideally should provide the following info:

content_type: (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type)
encoding: (see https://docs.python.org/2.4/lib/standard-encodings.html)
etag: (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag)

An optional dict "extras" is also provided for any other information.

Parameters:

Name	Type	Description	Default
`uri`	`str`	source uri to the resource	required
`chunk_size`	`int`	size for each chunk	required
`fileobj`	`IO[bytes]`	(IO[bytes]): temp fileobj for the stream	required

Exceptions:

Type	Description
`NotImplementedError`	[description]

Returns:

Type	Description
`Tuple[Iterable[bytes], iotoolz._abc.StreamInfo]`	Tuple[Iterable[bytes], StreamInfo]: tuple of the iterable and StreamInfo object describing the data

`readable(self)` ¶

`readline(self, size=-1)` ¶

Read and return one line from the stream. If size is specified, at most size bytes will be read.

Parameters:

Name	Type	Description	Default
`size`	`Optional[int]`	[description]. Defaults to -1.	`-1`

Returns:

Type	Description
`Union[str, bytes, bytearray]`	Union[str, bytes, bytearray]: [description]

`readlines(self, hint=-1)` ¶

Read and return a list of lines from the stream. hint can be specified to control the number of lines read: no more lines will be read if the total size (in bytes/characters) of all lines so far exceeds hint.

Parameters:

Name	Type	Description	Default
`hint`	`Optional[int]`	[description]. Defaults to -1.	`-1`

Returns:

Type	Description
`Union[List[str], List[bytes], List[bytearray]]`	Union[List[str], List[bytes], List[bytearray]]: [description]

`rmdir(self, ignore_errors=False, **kwargs)` ¶

Remove the entire directory.

`save(self, data=None, close=False)` ¶

Flush and stream everything in the buffer to the actual resource location.

Does nothing if mode is read-only. Will not close the stream.

Parameters:

Name	Type	Description	Default
`data`	`Union[bytes, bytearray, str]`	(Union[bytes, bytearray, str], optional): Write additional data to stream before saving. Defaults to None.	`None`
`close`	`bool`	Close stream after saving. Defaults to False.	`False`

Returns:

Type	Description
`AbcStream`	[AbcStream]: current stream object.

`seek(self, offset, whence=0)` ¶

Change the stream position to the given byte offset. offset is interpreted relative to the position indicated by whence. The default value for whence is SEEK_SET. Values for whence are:

SEEK_SET or 0 – start of the stream (the default); offset should be zero or positive

SEEK_CUR or 1 – current stream position; offset may be negative

SEEK_END or 2 – end of the stream; offset is usually negative

Return the new absolute position.

Parameters:

Name	Type	Description	Default
`offset`	`[type]`	offset relative to position indicated by "whence".	required
`whence`	`int`	reference position - 0, 1, or 2. Defaults to 0.	`0`

Returns:

Type	Description
`int`	[int]: new absolute position.

`seekable(self)` ¶

Whether the stream is seekable.

Returns:

Type	Description
`bool`	bool: whether stream is seekable.

`set_encoding(self, encoding)` ¶

Set and update the encoding to use to decode the binary data into text.

Parameters:

Name	Type	Description	Default
`encoding`	`str`	text encoding for the stream.	required

Returns:

Type	Description
`AbcStream`	AbcStream: current stream object.

`set_info(self, info=None, **kwargs)` ¶

`stats(self)` ¶

Retrieve the StreamInfo of the current stream.

`stats_(self)` ¶

Retrieve the StreamInfo.

`sync(self, force=False)` ¶

Loads the resource into buffer if not loaded.

Parameters:

Name	Type	Description	Default
`force`	`bool`	If true, load the resource into buffer again even if it is already loaded.	`False`

Returns:

Type	Description
`AbcStream`	AbcStream: current stream object.

`tell(self)` ¶

Return the current stream position.

Returns:

Type	Description
`int`	int: current stream position.

`unlink(self, missing_ok=True, **kwargs)` ¶

Delete and remove the resource.

`write(self, data)` ¶

Write the given bytes-like object into buffer and return the number of bytes written.

The data will not be written to the actual resource until "save" or "close" method is called.

Parameters:

Name	Type	Description	Default
`data`	`Union[str, bytes, bytearray]`	bytes-like object to write.	required

Exceptions:

Type	Description
`IOError`	stream is readonly.
`TypeError`	expect data to be of type str, bytes or bytearray.

Returns:

Type	Description
`int`	int: number of bytes written.

`write_from_fileobj_(self, uri, fileobj, size, **kwargs)` ¶

Abstract method to implement the write to the destination resource from the provided file-like object.

This file-like object only provides binary outputs - i.e. if the resources only accepts text input, you will need to decode the output inside this method.

Parameters:

Name	Type	Description	Default
`uri`	`str`	destination uri of the resource	required
`fileobj`	`IO[bytes]`	file-like object to read from	required
`size`	`int`	size of the data inside the file-like object	required

Exceptions:

Type	Description
`NotImplementedError`	[description]

Returns:

Type	Description
`StreamInfo`	StreamInfo: StreamInfo object describing the data

Last update: October 27, 2020

AbcStream¶

iotoolz._abc.AbcStream ¶

Attributes¶

closed: bool property readonly ¶

content_type: str property readonly ¶

encoding: str property readonly ¶

etag: str property readonly ¶

info: StreamInfo property readonly ¶

INMEM_SIZE: int ¶

size: int property readonly ¶

supported_schemas: Set[str] ¶

Methods¶

__init__(self, uri, mode='r', buffering=-1, encoding=None, newline=None, content_type='', inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs) special ¶

clone(self, uri=None, info=None, **kwargs) ¶

close(self) ¶

exists(self) ¶

flush(self) ¶

glob(self, pattern='*') ¶

is_dir(self) ¶

is_empty(self) ¶

is_file(self) ¶

iter_bytes(self) ¶

iter_dir(self) ¶

iter_dir_(self) ¶

mkdir(self, mode=511, parents=False, exist_ok=False) ¶

open(uri, mode='r', buffering=-1, encoding=None, newline=None, inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs) classmethod ¶

peek(self, size=None) ¶

pipe(self, sink, text_mode=False) ¶

read(self, size=-1) ¶

read_to_iterable_(self, uri, chunk_size, fileobj, **kwargs) ¶

readable(self) ¶

readline(self, size=-1) ¶

readlines(self, hint=-1) ¶

rmdir(self, ignore_errors=False, **kwargs) ¶

save(self, data=None, close=False) ¶

seek(self, offset, whence=0) ¶

seekable(self) ¶

set_encoding(self, encoding) ¶

set_info(self, info=None, **kwargs) ¶

stats(self) ¶

stats_(self) ¶

sync(self, force=False) ¶

tell(self) ¶

unlink(self, missing_ok=True, **kwargs) ¶

write(self, data) ¶

write_from_fileobj_(self, uri, fileobj, size, **kwargs) ¶

`iotoolz._abc.AbcStream` ¶

`closed: bool` `property` `readonly` ¶

`content_type: str` `property` `readonly` ¶

`encoding: str` `property` `readonly` ¶

`etag: str` `property` `readonly` ¶

`info: StreamInfo` `property` `readonly` ¶

`INMEM_SIZE: int` ¶

`size: int` `property` `readonly` ¶

`supported_schemas: Set[str]` ¶

`init(self, uri, mode='r', buffering=-1, encoding=None, newline=None, content_type='', inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs)` `special` ¶

`clone(self, uri=None, info=None, **kwargs)` ¶

`close(self)` ¶

`exists(self)` ¶

`flush(self)` ¶

`glob(self, pattern='*')` ¶

`is_dir(self)` ¶

`is_empty(self)` ¶

`is_file(self)` ¶

`iter_bytes(self)` ¶

`iter_dir(self)` ¶

`iter_dir_(self)` ¶

`mkdir(self, mode=511, parents=False, exist_ok=False)` ¶

`open(uri, mode='r', buffering=-1, encoding=None, newline=None, inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs)` `classmethod` ¶

`peek(self, size=None)` ¶

`pipe(self, sink, text_mode=False)` ¶

`read(self, size=-1)` ¶

`read_to_iterable_(self, uri, chunk_size, fileobj, **kwargs)` ¶

`readable(self)` ¶

`readline(self, size=-1)` ¶

`readlines(self, hint=-1)` ¶

`rmdir(self, ignore_errors=False, **kwargs)` ¶

`save(self, data=None, close=False)` ¶

`seek(self, offset, whence=0)` ¶

`seekable(self)` ¶

`set_encoding(self, encoding)` ¶

`set_info(self, info=None, **kwargs)` ¶

`stats(self)` ¶

`stats_(self)` ¶

`sync(self, force=False)` ¶

`tell(self)` ¶

`unlink(self, missing_ok=True, **kwargs)` ¶

`write(self, data)` ¶

`write_from_fileobj_(self, uri, fileobj, size, **kwargs)` ¶