kiwiPy: Robust, high-volume, messaging for big-data and computational science workflows

Martin Uhrin, Sebastiaan P. Huber

Research output: Contribution to journalJournal articleResearchpeer-review

9 Downloads (Pure)

Abstract

In this work we present kiwiPy, a Python library designed to support robust message based communication for high-throughput, big-data, applications while being general enough to be useful wherever high-volumes of messages need to be communicated in a predictable manner. KiwiPy relies on the RabbitMQ protocol, an industry standard message broker, while providing a simple and intuitive interface that can be used in both multithreaded and coroutine based applications. To demonstrate some of kiwiPy's functionality we give examples from AiiDA, a high-throughput simulation platform, where kiwiPy is used as a key component of the workflow engine.
Original languageEnglish
Article number2351
JournalThe Journal of Open Source Software
Volume5
Issue number52
DOIs
Publication statusPublished - 2020

Fingerprint Dive into the research topics of 'kiwiPy: Robust, high-volume, messaging for big-data and computational science workflows'. Together they form a unique fingerprint.

Cite this