Developers information

Contributing to Crappy

If you want to help developing Crappy with us, we’ll be more than happy to welcome you in the community ! Here you’ll find some practical information on how Crappy works under the hood, and a few guidelines for contributors.

If you brought an improvement to your own version of Crappy, and you think it’s worth sharing with the community, don’t hesitate to create a pull request on GitHub ! If you do so, please enforce the following rules :

• Follow PEP8 as much as possible, except for the indents that we chose to lower from 4 to 2 spaces for compactness.

• Use the Google style for docstrings. Please comment and document your code extensively, and update the source of the documentation if needed.

• Use relevant and meaningful titles and descriptions for your commits. Starting from v2.0.0, the rules described here should be used for commit messages.

The development branch of Crappy is called develop, and is the one on which you should commit. Starting from v2.0.0, the master branch is never directly committed to.

Technical description of Crappy

Note

This is a very simplified overview of how the module actually works. Only the main ideas are presented, and many technical aspects are omitted. Reading the code remains the only way to truly understand it !

Crappy is written as a pure-Python module, and is divided in a number of submodules. The breakout of the submodules follow the logical organization of the objects, according to their types. The different types of objects are presented in this section.

Blocks and Links

Regular Blocks and Links

The Blocks are base objects that have each a specific function and are instantiated by users to achieve a given overall behavior in their scripts. The Blocks can exchange information together through Links, with no restriction on the number of Links and the type of Blocks they connect.

Under the hood, every Block is a child of the base Block, which is itself a child of multiprocessing.Process. Each Block thus runs in its own separate process, which is the solution we chose for achieving an optimal performance of the module. The main downsides of this architecture are a high complexity, and potential difficulties to ensure a smooth termination of all the processes. A detailed description of the objects and strategies used to achieve a clean parallelization can be found in the next section.

As Blocks live each in a separate process, sharing data between each other is not straightforward. In Crappy, data can be sent from one Block to another only if they have first been linked by a Link. Behind each Link is a multiprocessing.Pipe, a low-level object that carries the data. In addition to instantiating the Pipe, the Link object also provides methods for the Block to use when sending data.

Camera-related Blocks

One major downside of the Pipes is that they can overflow, in which case data from the sender Block is simply discarded when trying to send it. This behavior is especially inconvenient for sending images (because they’re so large), so an alternative solution was chosen for image-processing Blocks. Instead of sharing the acquired images via Pipes, they are instead written by the sender Block to a shared Array where they can then be read by receiver processes. The receiver processes can display the images, record them, or process them.

Note that these receiver processes are not Blocks. Instead, they are children of the base CameraProcess, and are managed by a Camera Block in a quite similar way as the base Block manages all the other ones. Using this solution, we were able to parallelize the image acquisition, display, recording, and processing, which greatly improved the performance compared to previous versions of Crappy.

The use of shared Arrays to exchange data between Blocks was not chosen in the general case for several reasons. First, it adds an extra complexity that is not needed when sending numerical data. And second, it requires to know in advance the size of the data to share, which is easy to determine for images but not for numerical data.

Actuators, Cameras, InOuts

Some of the Blocks rely on specific types of helper object, that they can drive. It is the case for :

• The Camera Block that drives one Camera object for acquiring images.

• The IOBlock Block that drives one InOut object for acquiring data and/or setting outputs on hardware.

• The Machine Block that drives one or several Actuator objects for controlling motors and other actuators.

The Actuators, Cameras and InOuts are simple classes that do not derive from a parent class like the Blocks do. They were introduced to implement standardized ways for the Camera, IOBlock and Machine Blocks to interface with hardware. If written correctly. all the children of one of these classes implement the same methods and are seamlessly interchangeable.

In addition to providing a standardized way to integrate hardware in Crappy, these classes also provide helper methods to their children. For example, the InOut class implements a way to offset the inputs to zero before the test starts. Other example, the Camera class provides support for the integration of the supported camera settings in the CameraConfig window.

Modifier objects

The Modifier objects provide extra flexibility for fine-tuning the data flowing through the Links without having to modify or create Blocks. In practice, they are just callables (functions or classes) stored by a given Link and called each time data is sent through the Link. These objects are not meant to perform computationally-intensive tasks, as their call is not parallelized.

C++ extension modules

In the src folder of Crappy, you can find next to the module crappy another directory called ext. It contains the C++ extensions that were historically used by some objects in the module. It is very unsure whether these extensions still work, but they were kept around as a legacy waiting for pure-Python replacement solution to be added to Crappy. To enable extension module(s), one has to locally clone Crappy and install it manually with the correct drivers installed on the machine.

Other objects

Crappy is full of other helper objects, that have lower importance compared to the ones previously described and are not necessarily exposed to the users. Here is a non-exhaustive list of the main ones, and how they integrate in the framework.

Generator Paths objects

The Path objects are used by the Generator Block to create waveforms to send to downstream Blocks. Just like the InOuts for example, they standardize the methods of the Paths to make them interchangeable and implement convenient helper methods. The Paths are a bit less straightforward to use than the Actuators, Cameras and InOuts, and the possibility for users to create their own Paths was only recently added.

CameraConfig window

During the prepare() method of the Camera Block (or one of its children), the user can choose to enable the display of a CameraConfig window. This interactive tkinter based GUI allows to visualize the images acquired by the Camera object, and to interactively adjust the settings available for the Camera. All the code managing the configuration GUI is stored in the crappy.tool.camera_config submodule of Crappy. There, children of CameraConfig are defined to handle the specific needs of each child of the Camera Block. Also, helper classes are stored in separate files. The base CameraConfig is quite complex on its own, with a number of variables, bindings and traces that generate a feature-rich GUI. It even manages a parallel process (HistogramProcess) in which a histogram of the acquired images is calculated in real-time.

CameraSetting objects

To standardize the integration of the available settings for a given Camera object, the CameraSetting helper class was added to Crappy. It has three children that implement each a specific type of setting (boolean, integer/float, or choice from a given list). They manage the getter and the setter for the setting, as well as its integration in the CameraConfig window. In the base Camera object, the add_bool_setting(), add_scale_setting() and add_choice_setting() methods allow users to instantiate the desired settings. In addition, the add_trigger_setting() method provides specific support for the setting that manages the hardware trigger mode, if available on the camera. And finally, the add_software_roi() method manages the instantiation of 4 CameraScaleSetting at once, for applying a software ROI on the acquired images.

Image processing

The children of the Camera Block manage the execution of the various CameraProcess that might be requested by the user, including the one performing the image processing. The code performing the processing is however not included in the children of the Camera Block or the CameraProcess class. It is instead stored in a separate submodule, crappy.tool.image_processing. The rationale behind is to separate the code dealing with multiprocessing and the one performing image processing.

FT232H feature

While exploring the module, you will notice many occurrences of the term FT232H. It refers to a chip from FTDI, performing USB to I2C, SPI, Serial and GPIO conversion. It was integrated on one of Adafruit’s boards. We considered at some point the possibility to use it for achieving communication on low-level buses with Crappy, using only a PC and an FT232H. It turned out that the pyusb Python module required to talk to the chip is not process-safe, and a complex architecture had to be implemented to ensure multiprocess safety. This code can be found in the crappy.tool.ft232h submodule. For all the InOuts and Actuators communicating over low-level buses, a second version communicating through an FT232H was written and stored in the ft232h submodules.

After testing quite many options, we could not get the communication over FT232H to be completely stable. We always ended up with crashes, probably due to a wrong design of the server architecture used to ensure multiprocessing safety. In some cases though, the FT232H option worked really great and could be used on experimental setups without any problem. We thus decided to keep this feature in the module, but not to advertise it in the documentation and in the examples.

Detailed runtime sequence of Crappy

Crappy’s main strength lies in the use of massive parallelization to maximize the performance of the module. Unfortunately, this means we had to cope with Python’s notoriously complex multiprocessing architecture, and come up with a number of solutions to ensure a smooth execution and synchronization of all the processes. This section describes the different phases of Crappy’s execution, indicating for each phase which objects and mechanisms are involved and what they exactly do.

In the main Process

The __init__ phase

Before calling crappy.start() or crappy.prepare(), only one Process is running (the __main__ Process). All the instantiated Blocks will be children Processes of the __main__ Process, as soon as the next phase starts. The __main__ Process will normally live until the test is over and should not stop before any of its children.

As all Processes are children of __main__, it is a very natural position for __main__ to be the director managing all the other ones. And since all the instances of Block are meant to live in their own Process at some point, the methods required to drive all the Processes cannot be regular methods of Block. Instead, we have to rely on the classmethod of Block, because these methods will always be executed in __main__ if they are called in __main__. Moreover, because they operate at the class level, the classmethod are perfectly suited for managing the instances of Block. So, when reading the source code of the Block, remember that every classmethod is meant to be called directly from the __main__ Process and not by an instance of Block.

The first thing that happens in the Block when calling import crappy is that the class attributes of Block are initialized (mostly to None). These class attributes are multiprocessing synchronization objects used for managing the execution of all the Processes. They include :

• Two flags (bool) indicating whether all the Blocks have prepared and launched.

• A WeakSet storing the reference of all the instantiated Blocks.

• A list of all the names of the Blocks.

• An int specifying the minimum level for logging.

• A multiprocessing.Value storing the initial timestamp common to all the Blocks.

• A multiprocessing.Barrier used for ensuring that all the Blocks wait for each other before starting.

• Two multiprocessing.Event indicating the Blocks when to start and when to stop running.

• Two multiprocessing.Event signaling an Exception or a KeyboardInterrupt encountered by Crappy.

• A logging.Logger recording all the log messages from all the Blocks.

• A multiprocessing.Queue used for sending all the log messages to the Logger.

• A threading.Thread managing the execution of the Logger.

• A flag (bool) indicating the Logger Thread when to stop running.

• A flag (bool) indicating whether an Exception should be raised when Crappy terminates, in case one has been caught during Crappy’s execution.

Then, when a Block is instantiated, its instance attributes are initialized (mostly to None). Most of these instance attributes will later be set equal to the synchronization and logging class attributes. In addition to the synchronization and logging attributes, each instance of Block also has :

• A few attributes managing its execution (target looping frequency, niceness, flag for displaying the achieved looping frequency).

• A few buffers storing values needed for trying to achieve and displaying the looping frequency.

• A name, given by a classmethod to ensure it is unique.

Each instance of Block might of course also perform extra tasks, depending how the __init__ method of the child class is implemented. The __init__ phase ends when either crappy.start() or crappy.prepare() is called (the first thing start does is to call prepare).

The prepare phase

When the crappy.blocks.Block.prepare_all() classmethod (aliased to crappy.prepare() for conciseness) is called, it first sets the logging.Logger of the __main__ Process. Note that prepare_all() accepts one argument indicating the minimum level for logging. Then, all the synchronization class attributes listed above are instantiated to their target type (most of them were previously initialized to None). At that point, the number of Blocks is known, so the Barrier is set to this number +1 for the __main__ Process. The Value storing the initial timestamp is initialized to a negative value, to make it clear that it is not set yet.

Then, the USBServer Process tool is started if needed (see FT232H feature). After that, for each Block, its synchronization instance attributes are set to the corresponding class attributes of Block. Basically, the class attributes are shared with all the instances of Block. This is only possible because at that point the Blocks do not live in a separate Process yet, they all run in __main__. Short after, all the Blocks are started, meaning that they all run in a separate Process. If an exception is caught during the prepare phase, it first breaks the Barrier and then triggers The cleanup phase.

The renice phase

Right after the prepare phase should follow the renice phase. It corresponds to the call of the crappy.blocks.Block.renice_all() classmethod of the Block (aliased to crappy.renice() for conciseness). This method accepts one attribute, indicating whether negative nicenesses can be accepted (Linux and macOS only). On Windows, it does nothing as the concept of niceness is not defined. On Linux and macOS, it renices all the running Blocks to the value specified in their niceness attribute. Whether this value differs from default (0) depends on how the Blocks are written. If an exception is caught during the renice phase, it first breaks the Barrier and then triggers The cleanup phase.

The launch phase

The first thing happening after calling crappy.blocks.Block.launch_all() (aliased to crappy.launch() for conciseness) is that the __main__ Process starts waiting at the synchronization Barrier. This Barrier is shared by all the Blocks, and its value is set to the number of Blocks +1. Therefore, the Barrier only breaks when all the Blocks have reached it, as well as the __main__ Process. In case one of the Processes doesn’t make it to the Barrier, a BrokenBarrierError is raised to indicate all the other Blocks not to wait forever on the Barrier.

Once every Process has reached the Barrier, it breaks and releases them all. At that moment, the Value storing the initial timestamp is set to the current time (in seconds since epoch). After that, the start Event indicating all the Blocks to start looping is set, which releases them all. After that, the __main__ Process remains idle for most of the test, only waiting for one of the Blocks to finish. As soon as at least one Block is done, The cleanup phase starts. This phase also starts in case an Exception is caught.

The cleanup phase

This phase is triggered every time an exception (of any nature) is caught in the __main__ Process, or if at least one Block has stopped. The corresponding method is crappy.blocks.Block._exception(). Its goal is to make sure that all the Blocks stop as expected, and that the other Processes and Threads of Crappy terminate as well. It first sets the stop Event, indicating all the Blocks to stop looping and to finish as soon as possible. It then lets 3 seconds for all the Blocks to finish. If any Block is still alive passed this delay, it is mercilessly terminated. Then, the Process in charge of the USBServer is stopped, if applicable. Same goes for the Thread collecting all the log messages. Short before returning, Crappy is reset by the reset() method, that re-initializes all the synchronization objects. They are after all not needed anymore at that point. Finally, an exception might be raised in three cases :

• If all the Blocks are not done running at the end of this phase.

• If an Exception was caught during Crappy’s execution.

• If Crappy was stopped using Control-c, resulting in a KeyboardInterrupt.

The goal of this exception is to stop the execution of the __main__ Process, to avoid any more code to be executed in case something went wrong in Crappy. Note that this behavior can be disabled using the no_raise argument. In normal operating mode, if this phase ends without raising an exception, it indicates that Crappy executed and terminated gracefully.

In the children Processes

As soon as the start method of a Block is called, it starts running in a new Process separate from the __main__ one. It therefore lives it own independent life, and is only linked to the __main__ Process by the multiprocessing synchronization objects. The __main__ Process still has the option to kill the Blocks, if at the end of Crappy they do not stop by themselves.

When a Block is started, it firsts sets its Logger and runs its prepare() method to perform any preliminary task. Then, it reaches the Barrier, where it waits for all the other Blocks and the __main__ Process to be ready. If anything wrong happens before that, the Block breaks the Barrier, thus signaling its failure to the other ones through a BrokenBarrierError.

As soon as all the other Processes are ready, the Barrier breaks and releases the Block. This one then waits a second time for the __main__ Process to set the common start timestamp, after what all the Blocks are released. The begin() method is then called to perform any action specific to the first loop, and then the Block starts looping forever by calling it main() method. Under the hood, this method calls the loop() method, performing the main task for which the Block was written. It also handles the regulation and the display of the looping frequency, if requested by the user.

There are several ways the Block can stop. First, the stop Event might be set in another Process, which conducts each Block to stop running. Second, an Exception can be caught in the Block. And third, the Block might be killed by the __main__ Process if it becomes unresponsive. In the first two cases, the finish() method is called for performing the cleanup actions. The Block then stops running, and the associated Process finishes.