SeqTools: a python package for easy transformation, combination and evaluation of large datasets
SeqTools: une bibliothèque Python pour faciliter les transformations, combinaisons et l'évaluation de grands jeux de données
Résumé
SeqTools facilitates the manipulation of large datasets and the evaluation of a transformation pipeline. Some of the provided functionnalities include: mapping element-wise operations, reordering, reindexing, concatenation, joining, slicing, minibatching, etc... To improve ease of use, SeqTools assumes that dataset are objects that implement a list-like sequence interface: a container object with a length and its elements accessible via indexing or slicing . All SeqTools functions take and return objects compatible with this simple and convenient interface. Sometimes manipulating a whole dataset with transformations or combinations can be slow and resource intensive; a transformed dataset might not even fit into memory! To circumvent this issue, SeqTools implements on-demand execution under the hood, so that computations are only run when needed, and only for actually required elements while ignoring the rest of the dataset. This helps to keep memory resources down to a bare minimum and accelerate the time it take to access any arbitrary result. This on-demand strategy helps to quickly define dataset-wide transformations and probe a few results for debugging or prototyping purposes, yet it is transparent for the users who still benefit from a simple and convenient list-like interface. When comes the transition from prototyping to execution, the list-like container interface facilitates serial evaluation. Besides, SeqTools also provides simple helpers to dispatch work between multiple background workers (threads or processes), and therefore to maximize execution speed and resource usage. SeqTools originally targets data science, more precisely the preprocessing stages of a dataset. Being aware of the experimental nature of this usage, on-demand execution is made as transparent as possible to users by providing fault-tolerant functions and insightful error reporting. Moreover, internal code is kept concise and clear with comments to facilitate error tracing through a failing transformation pipeline. Nevertheless, this project purposedly keeps a generic interface and only requires minimal dependencies in order to facilitate reusability beyond this scope of application
Origine | Fichiers éditeurs autorisés sur une archive ouverte |
---|
Loading...