Skip to content

MinioStream

iotoolz.extensions.MinioStream is a stream interface implemented with AWS minio package.

By default, MinioStream will try to infer the credentials from the environment variables in the following order:

  • MINIO_ACCESS_KEY > AWS_ACCESS_KEY_ID
  • MINIO_SECRET_KEY > AWS_SECRET_ACCESS_KEY
  • MINIO_REGION > AWS_REGION > AWS_DEFAULT_REGION

NOTE

This is an extension module - i.e. you will need to pip install iotoolz[minio] before you can use this stream interface.

Unlike S3Stream the uri format should include the minio endpoint: minio://<endpoint>/<bucket>/<key>

iotoolz.extensions.minio.MinioStream

S3Stream is the stream interface to AWS S3 object store.

See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.S3Transfer

supported_schemas: Set[str]

Methods

__init__(self, uri, mode='r', buffering=-1, encoding=None, newline=None, content_type='', inmem_size=None, delimiter=None, chunk_size=8192, client=None, access_key=None, secret_key=None, region=None, secure=True, **kwargs) special

Creates a new instance of MinioStream.

See https://docs.min.io/docs/python-client-api-reference.html

Parameters:

Name Type Description Default
uri str

uri string to the resource.

 required
mode str

same as "open" - supports depends on the actual implementation. Defaults to "r".

 'r'
buffering int

same as "open". Defaults to -1.

 -1
encoding str

encoding used to decode bytes to str. Defaults to None.

 None
newline str

same as "open". Defaults to None.

 None
content_type str

mime type for the resource. Defaults to "".

 ''
inmem_size int

max size before buffer rollover from mem to disk. Defaults to None (i.e. never - may raise MemoryError).

 None
delimiter Union[str, bytes]

delimiter used for determining line boundaries. Defaults to None.

 None
chunk_size int

chunk size when iterating bytes stream. Defaults to io.DEFAULT_BUFFER_SIZE.

 8192
client Minio

overwrite the client to use for minio. Defaults to None.

 None
access_key str

minio access key (env MINIO_ACCESS_KEY). Defaults to None.

 None
secret_key str

minio secret key (env MINIO_SECRET_KEY). Defaults to None.

 None
region str

minio region (env MINIO_REGION). Defaults to None.

 None
secure bool

whether to use secure connection. (env MINIO_SECURE) Defaults to True.

 True

exists(self)

Whether the stream points to an existing resource.

is_dir(self)

Whether stream points to a existing dir.

is_file(self)

Whether stream points to a existing file.

iter_dir_(self)

Yields tuple of uri and the metadata in a directory.

mkdir(self, mode=511, parents=False, exist_ok=False)

This method does nothing as you do not need to create a 'folder' for an object store.

read_to_iterable_(self, uri, chunk_size, fileobj, **kwargs)

Downloads a minio object.

rmdir(self, ignore_errors=False, **kwargs)

Remove the entire directory.

stats_(self)

Retrieve the StreamInfo.

Delete and remove the resource.

write_from_fileobj_(self, uri, fileobj, size, **kwargs)

Uploads the data in the buffer.

Last update: January 8, 2021