refyre.cluster package

Submodules

refyre.cluster.Association module

class refyre.cluster.Association.AssociationCluster(starting_vals=None, mapper_function=<function AssociationCluster.<lambda>>, one_to_one=False, as_pathlib=False, input_vars=[])

Bases: object

GLOBAL_COUNTER = 0

all_clusters()

classmethod cleanup()

clusterify(mapper_function=None)

clusters = {}

item()

vals()

refyre.cluster.Broadcast module

class refyre.cluster.Broadcast.Broadcaster(*a, **kw)

Bases: object

classmethod add(id, cluster_vals)

classmethod clear()

files_dict = {}

ignore_list = []

classmethod release(cluster_id)

classmethod unlink(cluster_id)

refyre.cluster.Broadcast.insert(fdct, k, v)

refyre.cluster.Cache module

class refyre.cluster.Cache.AutoCache(cache_dir='.refyre_cache')

Bases: object

cache(var)

cache_files = {}

contains(fp)

decache(var, dest_dir='.')

load(fp)

classmethod obliterate(cache_instance)

Automatically wipe the entire cache object

save(pth=None)

refyre.cluster.Cluster module

class refyre.cluster.Cluster.FileCluster(input_paths=[], input_patterns=[], values=[], as_pathlib=False, recursive=False)

Bases: object

The workhorse of refyre. Variables do all of the heavylifting, harnessing the power of the data parsed through the fgraphs.

Paradigms - any method that involves transporting data must be wrapped in the refresher decorator method above. This will prevent overlapping variables from incorrectly altering results from each other, as well as ensures that variables only contain valid data.

GLOBAL_COUNTER = 0

all_clusters()

cache()

Saves all the files to cache

cached()

Returns all the FileCluster files in the cluster that are currently cached

classmethod cleanup()

clone()

The function returns a deep clone of the current object. :return: A deep clone of the current object is being returned.

clusters = {}

copy(target_dir, conflict_function=<function handle_file_conflict>)

Input:

• target_dir: the directory to which all files will be sent.

Returns

• FileCluster object with the new values

decache()

decompress(complete=False)

Takes any folders within the values, and replaces them with all the items in the folder

If complete is set to True, then it will recursively empty all files

delete(*args, **kwargs)

change_arr: List[Tuple(Str, Str)]

• Lists the Str that the path started out as, and what it became.

• If second tuple None, it implies a path was deleted

dirs()

Returns a FileCluster of all parent directories

filecount()

Returns the total number of files over all folders

filesize(format='bytes')

Returns the total file size, in bytes

filter(filter_func)

This function takes a filter function as input and returns a new FileCluster object with values filtered based on the input function.

Parameters:

filter_func – filter_func is a function that takes a filepath as input and returns a boolean

value. It is used to filter out values that we don’t want to keep in a FileCluster object. If the function returns True for a given filepath, it will be included in the filtered FileCluster object. If it :return: A FileCluster object with the filtered values.

item()

Returns a Pathlib.Paths list of all filepaths

map(map_func)

This function maps each filepath in a FileCluster object using a given function and returns a new FileCluster object with the mapped values.

Parameters:

map_func – map_func is a function that takes a filepath as input and returns a filepath

as output. This function is used to map each filepath in the values attribute of a FileCluster object. The resulting FileCluster object will have the same structure as the original one, but :return: A new FileCluster object with the mapped values.

move(*args, **kwargs)

change_arr: List[Tuple(Str, Str)]

• Lists the Str that the path started out as, and what it became.

• If second tuple None, it implies a path was deleted

post(url, additional_data, payload_name)

This function sends a post request to a specified API endpoint with additional payload parameters and returns the response object.

Parameters:

• url – The link to the API endpoint where the post request will be sent

• additional_data – a dictionary containing any additional metadata you want to send along with

the request. This could include things like authentication tokens or other parameters required by the API endpoint you are sending the request to :param payload_name: The name you want to give to the zip file that will be sent as part of the POST request :return: the Response object returned from the requests.post() call.

reduce(reducer_function)

The reduce function takes in a reducer function and returns the result of reducing the values in the array using the reducer function, or None if the array is empty.

Parameters:

reducer_function – A function that takes in two parameters and returns a third parameter

that can be used to reduce a list of values into a single value. This function is used in the reduce method of a class to perform the reduction operation on the list of values stored in the class instance :return: the result of the reduction of the values in the array using the given reducer function. If the array is empty, it returns None.

relink()

Reconnect the FileCluster to the broadcaster

rename(*args, **kwargs)

change_arr: List[Tuple(Str, Str)]

• Lists the Str that the path started out as, and what it became.

• If second tuple None, it implies a path was deleted

unlink()

Returns a FileCluster detached from the Broadcaster system

vals()

Returns a Pathlib.Paths list of all filepaths

classmethod wipe()

zip(save_dir='.', save_name='out.zip', conflict_function=<function handle_file_conflict>)

This function zips all the files in a directory into a single zip file.

Parameters:

save_dir – The directory where the zip file will be saved. If not specified, it will

default to the current working directory, defaults to . (optional) :param save_name: The name of the zip file that will be created, defaults to out.zip (optional) :param conflict_function: The conflict_function parameter is a function that is used to handle conflicts that may arise when trying to save the zip file to the specified directory. It takes in the path of the file that is being saved and returns a new path if there is a conflict with an existing file in the directory. If no :return: a FileCluster object that contains the path to the newly created zip file.

refyre.cluster.Cluster.handle_file_conflict(destination_path)

refyre.cluster.ClusterIterator module

class refyre.cluster.ClusterIterator.FileClusterIterator(variable)

Bases: object

A simple iterator to run through the variable data

Module contents