More about custom objects in Crappy
This last page of the tutorials covers various advanced topics related to the creation of custom objects in Crappy. Unlike for the three previous pages, the content of this fourth page will not be of interest for all users. It is still interesting to go over it for users wanting to have a deeper understanding of the module, or users with a specific need.
1. Custom Generator Paths
Starting from version 2.0.0, it is now possible for users to create their own Generator Paths ! There are two reasons why this possibility was added so late in the module. First, we’re not certain that there is a need for it. But since only a few modifications were needed to allow the creation of custom Paths, it was decided to make it possible anyway. And second, the implementation is a bit messier than for other custom objects. It should still be accessible for most users though, don’t worry !
Just like for the other custom objects, there is a template for creating
custom Paths and the Paths have to be children of
crappy.blocks.generator_path.meta_path.Path :
import crappy
class MyPath(crappy.blocks.generator_path.meta_path.Path):
def __init__():
super().__init__()
def get_cmd(self, data):
...
As you can see, there are only two methods to define ! Just like for the other
custom objects, __init__()
should initialize the parent class. It can also accept arguments, that will
correspond to the keys and values given in the dict passed to the
Generator Block. Note that in addition to these arguments, the value of
the last command sent by the Generator and the moment when it was sent are
accessible through the self.t0 and self.last_cmd attributes.
The get_cmd() method is for
generating the next command for the Generator to send. It must return the next
command as a float (None is also acceptable is there’s no new
command to send). It accepts one argument, which is the dict returned by
the recv_all_data() method of the Generator, and
that contains all the data recently received over incoming Links. It allows to
handle the case when Generator Paths have stop conditions based on the value of
a label, described in this tutorials section.
But how to handle the stop conditions ? And how to signal the Generator that a
stop condition was met ? This is where things get a bit trickier ! To indicate
that a stop condition is met, the
get_cmd() method simply has
to raise a StopIteration exception. That can be done anytime, based on
any arbitrary criterion. However, to make it so that conditions like
'delay=10' can be used, a
parse_condition() method is
provided by the base Path
class. It takes a str or a Callable or
None as its single argument, and always returns a Callable out of it.
This Callable accepts one argument, which is the dict that is passed as
an argument to get_cmd(),
and it returns a bool indicating whether the stop condition is met or
not.
So, to summarize, if your custom Path does not accept a 'condition' or
equivalent argument, you’re free to raise StopIteration whenever you
want to switch to the next Path based on arbitrary criteria. If you do have a
'condition' or equivalent argument, you should first parse it during
__init__() using the
parse_condition() method. It
will output a Callable, that you should store as a variable. Then, in the
get_cmd() method, you should
call this variable with the dict from
recv_all_data() as an argument. If it returns
True the condition is met and you should raise StopIteration.
Otherwise, you should return a value for the Generator to send.
It is definitely not the most straightforward implementation, but it is very flexible and should fit most situations. Let’s write a short example to make it clearer how to create a custom Generator Path and how to handle the conditions. This example generates a square wave, whose duty cycle can be either fixed or controlled by the value of an input label :
Note
To run this example, you’ll need to have the matplotlib and scipy
Python modules installed.
This example contains all the ingredients described above. The parent class is
initialized, then the condition argument is parsed with
parse_condition(). In
get_cmd(), the given
condition is checked based on the latest received data from upstream Blocks,
and raises StopIteration if needed. This method also returns
float values as expected, and the t0 attribute is used for
calculating the value to return.
The exact way the custom Path works won’t be detailed here, but it should be
self-explanatory by just reading the code and the comments. You can
download this custom Path example to run it locally on your
machine. You should see that the duty cycle of the generated square signal
varies according to the target duty cycle, as expected. In the custom objects
examples on GitHub, you’ll find another example of a custom
Generator Path.
Note
If you want to have debug information displayed in the terminal from your
Path, do not use the print() function ! Instead, use the
log() method provided by
the parent Path class. This
way, the log messages are included in the log file and handled in a nicer
way by Crappy.
There’s one more very specific point that we’d like to outline about the use of
Generator Paths in Crappy. Earlier, it was mentioned that the
parse_condition() method of
the base Path object accepts Callable. More precisely,
it accepts Callables that take as only argument a dict whose keys are
str and values are list, and that return a bool value.
This means that it is actually possible to pass a Callable as the value for
the condition argument, not just a str or None ! This
possibility is not often used, but at least you now know that it exists ! It
could for instance come in use if you want to use an existing Path, but you
have an unusual stop condition (e.g. one that depends on the values of two
labels).
2. More about custom InOuts
In addition to what was described in the tutorial section about how to
create custom InOut objects, there is one more minor
feature that the In / Out possess and that is worth describing in the
tutorials. That is the ability for an InOut to acquire data before a test
starts, and to use this data to offset the channels to zero. To do so, the
script must match two conditions. First, the make_zero_delay argument of
the IOBlock must be set to a positive value. And second, the used InOut
must have its get_data() method defined (it cannot be
a pure stream class). If both of these conditions are met, then the InOut will
acquire data using get_data() during
prepare() for the specified delay, and create
offsets so that for each acquired channel its value starts from zero at the
beginning of the test. It also works for streams, provided that the number of
channels acquired in streamer mode is the same as the number of channels
acquired by get_data().
Things get a bit trickier when the hardware can handle and tune offsets for
its channels ! In such a case, it might be advantageous to set the zeroing
offsets directly on the device rather than relying on Crappy. To achieve that,
the make_zero() method of the base
InOut has to be overriden in the child InOut class, and
the way it is performed depends on the capabilities of the hardware. What is
usually done is that the make_zero() method of the
base class calculates the offset values, and the one of the child class sets
these values on the hardware and resets the offsets on Crappy’s side. This
kind of implementation can be found in the Labjack T7 or the
Comedi InOuts. Check their code to see how it looks ! There is also a
very basic example of offsetting in the examples on GitHub where the method is overriden and the offsets are simply
doubled.
There is no need for a specific example in this sub-section, it is mostly included to signal the existence of the zeroing feature and the possibility for users to override it.
3. More about custom Actuators
In the tutorial section about how to create custom Actuator objects, then entire speed management aspect in position
mode was left out. In this section, we’re going to cover in more details
the possibilities for driving the speed in position mode, and how
to write a set_position() method
accordingly.
In the dict containing information about the
Actuator to drive, there are two optional keys that
allow tuning the target speed in position mode. They can both be set, or
only one, or none. These keys are :
'speed', that sets a target speed value from the beginning of the test. This value might be overriden if'speed_cmd_label'is given. If it is not overriden, it persists forever.'speed_cmd_label', that provides the name of a label carrying the target speed values. As soon as a value is received over this label, the previous target value is overriden and the new one is set.
If no target speed value is set, i.e. if none of the two possible keys is
provided or if 'speed' is not set and no target speed has been received
over the 'speed_cmd_label' so far, the target speed is set to
None.
Now, how is that reflected on your code when creating a custom Actuator ?
First, note that it only influences the
set_position() method, all the other ones are
unaffected. The target speed value is always passed to the Actuator as the
second argument of the set_position() method.
It is passed no matter its value, so it might be equal to None ! It is
your duty to handle the two situations when it has or hasn’t an actual value.
For hardware that doesn’t support speed adjustment when operated in position
mode, this argument can always be ignored. You can have a look at the
Actuators distributed with Crappy to see how
the various set_position() methods implement
the speed management in position mode. Also, an example of a Machine
Block with a variable target speed can be found in the blocks examples folder
on GitHub.
4. More about custom Cameras
Because image acquisition is such a complex topic, the
Camera object is by far the richest of the classes
interfacing with hardware in Crappy. For that reason, not all of its features
could be presented in the previous tutorial sections. The missing ones are
introduced here instead. Note that they are clearly secondary compared to the
other features already presented !
4.a. Pre-defined settings
4.a.1. Trigger setting
On the previous page, the three methods allowing to
instantiate a CameraSetting
were presented. While these methods cover a wide range of situations, we found
that they were not always well-suited to manage the trigger setting that some
cameras possess. Indeed, when a camera is switched to external trigger mode, it
will only acquire images when receiving an external signal. But if this signal
is itself issued by a device controlled from Crappy, then the camera cannot
acquire images for display in the
CameraConfig window, as the
InOut used for generating the signal will only do so
once the configuration window closes ! To address this problem, a new
method was introduced specifically for instantiating a trigger setting :
the add_trigger_setting() method !
When calling this method, a new
CameraChoiceSetting is
instantiated with the name 'trigger'. Its possible choices are
'Free run', 'Hdw after config' and 'Hardware', and its
default is 'Free run'. The only arguments left for the user to set are
thus the getter and the setter methods. This trigger setting appears in the
configuration window just like any other setting, and can be accessed and
modified in the code as well. It is really just a normal setting, but with a
pre-determined name and choices !
When set to 'Free run' mode, the camera should acquire images without
needing an external trigger. When set to 'Hardware', the camera should
only acquire images when receiving a hardware trigger. What is more interesting
is definitely the 'Hdw after config' mode : when set, the camera stays in
free run mode as long as the configuration window is opened, but switches to
hardware trigger mode as soon as the window is closed ! This way, you can
adjust the various settings interactively in the configuration window, but
still use the hardware trigger mode for the test !
As mentioned above, the user still has to define the getter and setter methods.
For the setter, both the 'Free run' and 'Hdw after config' settings
should set the camera to free run mode, and the 'Hardware' setting should
set the camera to hardware trigger mode. For the getter now, it should return
'Hardware' is the camera is in hardware trigger mode, and either
'Free run' or 'Hdw after config' otherwise, depending on the last
value set by the setter. It is not the most straightforward getter to
implement, we know ! This aspect should be improved in future releases, but for
now you’ll have to cope with it. You can get inspiration from the Xi API
Camera that implements it already.
4.a.2. Software ROI setting
In addition to the trigger setting, another improvement was brought to make
user’s life easier : the add_software_roi() method.
It allows to crop the acquired images to the desired dimension, so that
they take less space when recorded, or can be processed faster. The remaining
Region Of Interest should of course only contain the area relevant to your
test. Unlike the hardware ROI setting that some cameras might possess, this
setting does not influence the image acquisition, and thus does not improve the
acquisition rate.
Under the hood, the add_software_roi() method
instantiates four
CameraScaleSetting managing
the position and size of the ROI. These settings are 'ROI_x',
'ROI_y', 'ROI_width' and 'ROI_height', and their arguments
are inaccessible to the user. The only values that the user has to provide are
the width and the height of the acquired images, as arguments to the
add_software_roi() method.
The application of the software ROI to the acquired images is not automatic,
you have to run the apply_soft_roi() on the
acquired image in order for it to be effective. It returns the cropped image,
or None if there’s nothing left to display (shouldn’t happen). You can
find examples of usage for the software ROI in
CameraOpencv, or in the examples folder on GitHub.
4.b. Reload slider and choice settings
The software ROI setting described in the previous sub-section sure is nice,
but what happens to it when the size of the acquired images change because of
another setting that controls the image format ? After all, the limits of the
sliders that it creates depend on the image size given by the user, and once
the open() method of Camera
returns, there’s no way to re-instantiate the settings. To address this
problem, and all the similar ones that users might face, we added the
possibility to “reload” the
CameraScaleSetting and the
CameraChoiceSetting.
Reloading a setting means either adjusting the limits of the slider, or
changing the labels and/or the number of choices, depending on the type of
setting.
In practice, each setting (except for the boolean ones) possess a
reload() method,
that allows to reload it. The arguments to provide depend on the type of
setting. The calls to
reload() should
be placed in the relevant getter or setter methods, so that when the value of a
setting changes it adjusts the other settings accordingly. It is totally not
mandatory to do so, and most Cameras won’t ever need to reload any setting. For
the specific case of the software ROI setting, the
Camera class defines a specific
reload_software_roi() method for reloading it. You
can check the CameraOpencv Camera to see an example of
a class implementing a setting reload.
Important
The possibility to reload settings is still recent, and might not be fully stable. If you have trouble using it, please report it (see the Troubleshooting page).
4.c. Manage the metadata of the images
For the last feature of th Camera objects presented in
the tutorials, let’s introduce the possibility to include metadata in the
information returned by a Camera ! So far, it was always mentioned that the
first value that the open() method of Cameras
should return is the timestamp of the acquired image. That is actually
incorrect, since it is also possible to return a dict containing
metadata about the acquired image ! This option is only interesting if the
used camera can return metadata, such as the frame number, the aperture, the
exposure time, etc.
The returned dictionary should replace the bare timestamp value, and must
contain at least two keys. The 't(s)' key contains the timestamp of the
image, as given by the time.time. And the 'ImageUniqueID' key
should contain an integer allowing to identify the image, like the index of
the acquired frame. In the case when only a timestamp is returned (and not a
metadata dict), the frame index is calculated automatically by Crappy
based on the images it sees, but might not correspond to the real frame index
of the camera.
Apart from these two mandatory keys, the user is free to include any other key
carrying any other type of information. Relevant information in the context of
experimental research could be the moment when the image was captured
(different from the moment when it was transmitted to Crappy), the exposure
time, etc. All the data included in the returned dictionary is meant to be
written in a metadata.csv file saved along with the recorded images, that
contains for each image its metadata. For each key of the dictionary that is a
valid EXIF tag, the metadata will also be embedded in the recorded images if
the PIL backend is used for recording. The 'ImageUniqueID' is
already a valid EXIF tag, and the time information is split and recorded over
the 'DateTimeOriginal' and 'SubsecTimeOriginal' tags. For now, none
of the Cameras implemented in Crappy return metadata as a dict, but that
will change in future releases !
5. Custom Camera Blocks
On the previous tutorial page, a section was
dedicated to the instantiation of custom Blocks. Always moving one step
further into customization, we’re going to see in this section how you can
create your own subclass of a particular subclass of Block, namely the
Camera Block !
Basically, the Camera Block provides three functionalities. First, it acquires images by driving a Camera object. Then, it can optionally display the acquired images in a dedicated window. And third, it can optionally record the acquired images. The great advantage of this Block is that it can perform these three operations in parallel, and therefore optimize the framerate for each functionality. The counterpart is that these three operations must be embedded into a single Block, rather than performed separately by three different Blocks. More details about the implementation of the Camera Block can be found in the Developers section of the documentation.
In the Camera Block, some lines of code provide the possibility to perform a fourth operation in parallel : image processing on the acquired images. While the Camera Block itself does not make use of this possibility, children of Camera can use it very easily and implement parallelized image processing. For instance, the Video Extenso and the DIC VE Blocks are children of Camera that implement real-time video-extensometry on the acquired images. So, in most cases, the Camera Block should be subclassed by users wishing to implement their own custom image processing method in Crappy.
Now, in practice, how to write your own subclass of Camera ? As mentioned
above, the base Camera Block already handles the acquisition, the display, and
the recording of the images. All that’s left for you to define is how to
correctly process the images, and what results to send to downstream Blocks.
But remember that just like the other functionalities, the processing is also
parallelized ! This means that it cannot be performed directly in the custom
Camera Block, but rather in another object : a
CameraProcess. So, anyone who wants to
implement their own image processing in Crappy must create two new classes :
one child of Camera, and one child of
CameraProcess !
5.a. The CameraProcess class
Just like the other custom objects that you can instantiate in Crappy, there is
a template for the CameraProcess :
import crappy
class MyCameraProcess(crappy.blocks.camera_processes.CameraProcess):
def __init__():
super().__init__()
def init(self):
...
def loop(self):
...
def finish(self):
...
Let’s review one by one the methods that you can define :
In
__init__()you should only handle the arguments that your CameraProcess accepts, nothing more ! The reason for that is that this method runs in a separate “context” than the following ones, so as little as possible should be performed there.init()is where you can instantiate and initialize the various objects that you will use for the image processing. It is fine to leave this method undefined.loop()is called repeatedly, and is the equivalent of theloop()method of the Block. It should handle the received images, process them, and send the result to downstream Blocks. The methods and objects to use for that are detailed below.finish()is the equivalent of thefinish()method of the Block. It is called at the very end when Crappy finishes, and should de-initialize the objects used for the image processing. It is fine to leave this method undefined.
The Base CameraProcess class handles the calls to these methods, as well as the exceptions that might be raised. All the user has to do is to define them. In addition to the methods that the user has to define, there are three other methods that can be called and provide extra functionalities :
send()is the equivalent of thesend()method of the Block, of which it is almost an exact copy. It allows to send data to downstream Block, and takes one argument either as adictor as anIterableif theself._labelsattribute is defined (and notself.labelslike in the Block). Refer to the method of Block for more information.send_to_draw()allows to sendOverlayobjects for the displayer to show as an overlay on top of the displayed images. It is discussed in more details in a next subsection.log()is the equivalent of thelog()method of the Block, and allows handling log messages without resorting to theprintfunction.
On top of that, two very useful attributes are defined by the CameraProcess class :
self.imgcontains the latest image captured by the Camera Block, as anumpyarray. It is updated automatically, so users just have to use it as is. Also note that theloop()method is only called again if a new image was received since the last call, soself.imgshould be a different image at every call !self.metadatacontains the metadata associated with the image stored inself.img. The metadata is in the format described in the dedicated section. It is especially useful for retrieving the timestamp and the frame index of the processed image.
Now that you have a general overview of the methods and attribute that the CameraProcess exposes, it is time to demonstrate how to use them in a demo CameraProcess :
# coding: utf-8
import crappy
import cv2
class CustomCameraProcess(crappy.blocks.camera_processes.CameraProcess):
def __init__(self,
scale_factor=1.2,
min_neighbors=3):
super().__init__()
self._eye_cascade = None
self._scale_factor = scale_factor
self._min_neighbors = min_neighbors
def init(self):
self._eye_cascade = cv2.CascadeClassifier(
cv2.data.haarcascades + 'haarcascade_eye.xml')
def loop(self):
eyes = self._eye_cascade.detectMultiScale(self.img,
scaleFactor=self._scale_factor,
minNeighbors=self._min_neighbors)
self.send({'t(s)': self.metadata['t(s)'], 'eyes': eyes})
In the example code, the defined class uses OpenCV to detect eyes on the
received images. It returns the timestamp of the image, and an object
containing the coordinates of the detected eyes. Here, the
finish() method is missing,
because there is nothing to de-initialize. As described above, the
__init__() method only
handles the given arguments,
init() makes the class
ready for looping, and
loop() performs the main
detection task. The self.img attribute is used as an argument to the eye
detection function, and self.metadata is used for returning the timestamp
of the current image to downstream Blocks. This class alone is not enough for
running the eye detection with Crappy, a corresponding custom
Camera Block now has to be defined in the next
subsection !
Note
By default, the loop()
method is called every time a new image is grabbed by the CameraProcess. It
is possible to tune this behavior by overriding the
_get_data() method. See
the ImageSaver Process for an
example.
Note
By default, a counter accessible via the self.fps_count attribute is
incremented every time a new image is grabbed by the CameraProcess. It is
only used in case the display_freq argument of the
Camera Block is set to True, to keep track of
the framerate achieved by the CameraProcess. If you have specific situations
to handle (e.g. a call to
loop() that does not
actually process the new image), you can access the self.fps_count
attribute and decrement or modify it yourself.
5.b. Writing the custom Camera Block
To be able to use your freshly defined custom
CameraProcess, you now have to create
a custom Camera Block that makes use of the
CameraProcess. Since most of the complexity is handled in the base parent
class, the template for a child of the Camera Block is pretty basic :
import crappy
class MyCameraBlock(crappy.blocks.Camera):
def __init__(self,
camera,
transform=None,
config=True,
display_images=False,
displayer_backend=None,
displayer_framerate=5,
software_trig_label=None,
display_freq=False,
freq=200,
debug=False,
save_images=False,
img_extension="tiff",
save_folder=None,
save_period=1,
save_backend=None,
image_generator=None,
img_shape=None,
img_dtype=None,
**kwargs):
super().__init__(camera=camera,
transform=transform,
config=config,
display_images=display_images,
displayer_backend=displayer_backend,
displayer_framerate=displayer_framerate,
software_trig_label=software_trig_label,
display_freq=display_freq,
freq=freq,
debug=debug,
save_images=save_images,
img_extension=img_extension,
save_folder=save_folder,
save_period=save_period,
save_backend=save_backend,
image_generator=image_generator,
img_shape=img_shape,
img_dtype=img_dtype,
**kwargs)
def prepare(self):
self.process_proc = CustomCameraProcess()
Notice that since your new __init__() method
overrides the one from the parent class, you have to handle all the parameters
of the parent class in addition to the ones that you might add ! As usual,
__init__() should instantiate all the objects that
will be used in your class and handle the arguments. In simple cases,
prepare() is very basic and is only used for
setting the CameraProcess to use. Except for that, there is nothing more to
do on the Camera Block side !
Note
If you use the VideoExtenso Block for example, you
have to select spots to track in the configuration window. To achieve such
a behavior, you’ll need to override the
_configure() method in your child Camera Block,
and to define your own version of
CameraConfig. This possibility is very
specific, so it is not described in the tutorials.
5.c. Sending an overlay to the Displayer
Because the CameraProcess deals with
images, it can be interesting to have a real-time display of how the processing
is performing. To do so, the base CameraProcess class provides the
send_to_draw() method that
allows to send objects to the
Displayer Process to draw overlays on
top of the displayed images. Of course, it will only work if the
display_images argument of the Camera Block is set to True.
The objects indicating what to draw should be children of the
Overlay class. They only need
to define the draw()
method, that takes the image to display as an argument and draws the overlay on
top of it. Here is what it looks like for displaying a black ellipse :
# coding: utf-8
import crappy
import cv2
class Ellipse(crappy.tool.camera_config.Overlay):
def __init__(self, center_x, center_y, x_axis, y_axis):
super().__init__()
self._center_x = center_x
self._center_y = center_y
self._x_axis = x_axis
self._y_axis = y_axis
def draw(self, img):
thickness = max(img.shape[0] // 480, img.shape[1] // 640, 1) + 1
cv2.ellipse(img,
(self._center_x, self._center_y),
(self._x_axis, self._y_axis),
0,
0,
360,
0,
thickness)
To transmit the overlay to the Displayer Process, the
send_to_draw() should send
a collection of instances of Overlays. It is as simple as that ! Crappy only
comes with one predefined Overlay object, the
Box, but it is easy enough to
define your own ones. Here is what the custom CameraProcess defined in the
previous sub-section looks like after integrating the code for sending
overlays :
# coding: utf-8
import crappy
import cv2
class CustomCameraProcess(crappy.blocks.camera_processes.CameraProcess):
def __init__(self,
scale_factor=1.2,
min_neighbors=3):
super().__init__()
self._eye_cascade = None
self._scale_factor = scale_factor
self._min_neighbors = min_neighbors
def init(self):
self._eye_cascade = cv2.CascadeClassifier(
cv2.data.haarcascades + 'haarcascade_eye.xml')
def loop(self):
eyes = self._eye_cascade.detectMultiScale(self.img,
scaleFactor=self._scale_factor,
minNeighbors=self._min_neighbors)
to_draw = list()
for (x, y, width, height) in eyes:
to_draw.append(Ellipse(int(x + width / 2), int(y + height / 2),
int(width / 2), int(height / 2)))
self.send_to_draw(to_draw)
self.send({'t(s)': self.metadata['t(s)'], 'eyes': eyes})
5.d. Final runnable example
It is now time to put together all the custom classes that were defined in the
previous sub-sections. There is first the custom
Overlay class for drawing an
ellipse overlay on top of the displayed images. It is used by the custom
CameraProcess that performs eye
detection on the acquired images. This custom CameraProcess is itself
instantiated by a custom child of the Camera Block,
that is the final object called by the user in its script. Based on these
development, here is a final runnable code performing eye detection and adding
the detected eyes on the displayed images :
Note
To run this example, you’ll need to have the opencv-python and Pillow Python modules installed.
This custom Camera Block script is based on an example that you can find in the
custom objects examples folder on GitHub. You
can download it to run it locally
on your machine. Note that the 'Webcam' camera is used here, so this
example will require a camera readable by OpenCV to be plugged to the computer.
The instantiation of custom image processing in Crappy is definitely one of the
most advanced things you can perform, but it is totally worth it if you want to
have your processing parallelized with the acquisition and the display and/or
recording of the images. There will likely be changes and improvements on these
aspects in future releases.