TempStream¶
iotoolz.TempStream is a stream interface implemented with python tempfile package.
iotoolz.temp.TempStream
¶
TempStream is the stream interface to an in-memory buffer with can rollover to local file system if the "inmem_size" arg is set.
supported_schemas: Set[str]
¶
Methods¶
__init__(self, uri, data=None, mode='r', buffering=-1, encoding=None, newline=None, content_type='', inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs)
special
¶
Creates a new instance of TempStream.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
uri |
str |
uri string to the resource. |
required |
data |
Union[str, bytes, bytearray] |
initial data to the stream. |
None |
mode |
str |
same as "open" - supports depends on the actual implementation. Defaults to "r". |
'r' |
buffering |
int |
same as "open". Defaults to -1. |
-1 |
encoding |
str |
encoding used to decode bytes to str. Defaults to None. |
None |
newline |
str |
same as "open". Defaults to None. |
None |
content_type |
str |
mime type for the resource. Defaults to "". |
'' |
inmem_size |
int |
max size before buffer rollover from mem to disk. Defaults to None (i.e. never - may raise MemoryError). |
None |
delimiter |
Union[str, bytes] |
delimiter used for determining line boundaries. Defaults to None. |
None |
chunk_size |
int |
chunk size when iterating bytes stream. Defaults to io.DEFAULT_BUFFER_SIZE. |
8192 |
etag |
str |
etag for the stream content. Defaults to "". |
'' |
exists(self)
¶
TempStream will always exist.
iter_dir(self)
¶
If the current stream is a directory, this method will yield all Stream in the directory. Otherwise, it should yield all Stream in the same directory (or level) as the current stream.
iter_dir_(self)
¶
If the current stream is a directory, this method should yield StreamInfo in the directory. Otherwise, it should yield other StreamInfo in the same directory (or level) as the current stream.
mkdir(self, mode=511, parents=False, exist_ok=False)
¶
This method does nothing as you do not need to create a folder for an in-memory buffer.
open(uri, mode='r', buffering=-1, encoding=None, newline=None, inmem_size=None, delimiter=None, chunk_size=8192, etag='', **kwargs)
classmethod
¶
Creates a new instance of TempStream.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
uri |
str |
uri string to the resource. |
required |
mode |
str |
same as "open" - supports depends on the actual implementation. Defaults to "r". |
'r' |
buffering |
int |
same as "open". Defaults to -1. |
-1 |
encoding |
str |
encoding used to decode bytes to str. Defaults to None. |
None |
newline |
str |
same as "open". Defaults to None. |
None |
content_type |
str |
mime type for the resource. Defaults to "". |
required |
inmem_size |
int |
max size before buffer rollover from mem to disk. Defaults to None (i.e. never - may raise MemoryError). |
None |
delimiter |
Union[str, bytes] |
delimiter used for determining line boundaries. Defaults to None. |
None |
chunk_size |
int |
chunk size when iterating bytes stream. Defaults to io.DEFAULT_BUFFER_SIZE. |
8192 |
etag |
str |
etag for the stream content. Defaults to "". |
'' |
**kwargs |
|
Additional keyword arguments to the stream (depends on implementation) |
{} |
Returns:
| Type | Description |
|---|---|
TempStream |
TempStream: new instance of TempStream |
read_to_iterable_(self, uri, chunk_size, fileobj, **kwargs)
¶
read_to_iterable_ is an abstract method to implement the reading of the source resource into the a binary iterator - i.e. you will need to encode your data appropriately if it is a string.
Alternatively, 'fileobj' argument is also provided where you can write to the stream file buffer directly. In this case, you should return an empty iterable. However, this will be less efficient as the actual read will only start after all the data have been read into the buffer.
The method should return a tuple of the bytes iterator and StreamInfo object which ideally should provide the following info:
-
content_type: (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type)
-
encoding: (see https://docs.python.org/2.4/lib/standard-encodings.html)
-
etag: (see https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag)
An optional dict "extras" is also provided for any other information.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
uri |
str |
source uri to the resource |
required |
chunk_size |
int |
size for each chunk |
required |
fileobj |
IO[bytes] |
(IO[bytes]): temp fileobj for the stream |
required |
Exceptions:
| Type | Description |
|---|---|
NotImplementedError |
[description] |
Returns:
| Type | Description |
|---|---|
Tuple[Iterable[bytes], iotoolz._abc.StreamInfo] |
Tuple[Iterable[bytes], StreamInfo]: tuple of the iterable and StreamInfo object describing the data |
stats_(self)
¶
Retrieve the StreamInfo.
unlink(self, missing_ok=True, **kwargs)
¶
Delete and remove the resource.
write_from_fileobj_(self, uri, fileobj, size, **kwargs)
¶
Abstract method to implement the write to the destination resource from the provided file-like object.
This file-like object only provides binary outputs - i.e. if the resources only accepts text input, you will need to decode the output inside this method.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
uri |
str |
destination uri of the resource |
required |
fileobj |
IO[bytes] |
file-like object to read from |
required |
size |
int |
size of the data inside the file-like object |
required |
Exceptions:
| Type | Description |
|---|---|
NotImplementedError |
[description] |
Returns:
| Type | Description |
|---|---|
StreamInfo |
StreamInfo: StreamInfo object describing the data |