Derivatives¶
Derivatives are files, usually generated automatically from a source file, which may be useful to the repository. Examples of derivatives include:
- smaller or compressed service files
- thumbnails or poster images for display
- preservation files in open file formats
- files containing technical metadata about a file.
Derivative Models¶
There are two schemes you can use for derivatives:
- In the standard model, each derivative is a new Media entity linked to the original's parent node. The "Media Use" field is used to distinguish the roles that the various files and media have (Thumbnail, Service File, etc.).
- In the multi-file media model, derivatives are added to additional file fields on the original file's Media entity (see Multi-File Media).
Derivatives are Actions in Drupal¶
Derivative configuration is stored using Drupal's Actions.
This means that all derivative configuration, such as
parameters dictating the derivative
size and quality can be edited by a repository administrator
in the Drupal GUI under Manage > System > Actions.
As Actions, they can be executed on nodes manually using Views Bulk Operations. They can also be configured to run automatically on media save thanks to Islandora's additions to the Drupal [Contexts] module.
Derivative actions will replace existing files
If a derivative action runs but the target derivative (as identified by its taxonomy term) already exists, the new file will replace the old file (leaving the rest of the Media intact).
Derivatives can run automatically with Contexts¶
With the Contexts module, you can configure derivatives to run under specific conditions. These are set in the "Conditions" section. Islandora provides a number of context conditions including:
- entity bundle
- media mimetype
- term (attached to a media or node, or a node's parent node)
- whether a node or media "is islandora"
You can set up as many of these as you like, with "and" or "or" logic between them.
In the "Reactions" section is where you can set up derivatives. For standard model derivatives, choose the "Derivatives" reaction. This lists all actions, including derivative actions. Note that multifile derivatives won't work here.
Multi-file media
The multi-file media derivatives can NOT be selected from within the "Derivatives" reactions. From the "Reactions" pop-up window, you must choose "Derive file for Existing Media". This panel lists only Multi-file media-type derivatives.
Derivatives have Types¶
When creating a new Derivative Action, there are a number of flavours of derivative "types" available. All derivative Actions fall into one of these types.
| Derivative type name | machine name | Supplying module | Expected Microservice (software) |
|---|---|---|---|
| Generate a Technical metadata derivative | generate_fits_derivative |
Islandora FITS (roblib) | Crayfits (FITS) |
| Generate a audio derivative | generate_audio_derivative |
Islandora Audio | Homarus (FFmpeg) |
| Generate a video derivative | generate_video_derivative |
Islandora Video | Homarus (FFmpeg) |
| Generate an image derivative | generate_image_derivative |
Islandora Image | Houdini (ImageMagick) |
| Get OCR from image | generate_ocr_derivative |
Islandora Text Extraction | Hypercube (tesseract/pdftotext) |
Multi-file media
The derivatives types available for multi-file media are the ones marked as "for Media Attachment" e.g. "Generate an Image Derivative for Media Attachment".
Derivatives are created by microservices ("Crayfish")¶
In Islandora, we generate derivatives using microservices, which are small web applications that do a single task. Each microservice takes the name of a file as well as some parameters. It runs an executable and returns a transformed file, which can be loaded back into the repository. The microservices in Islandora stack are:
| Repository | Microservice name | executable |
|---|---|---|
| Crayfish | Homarus | FFmpeg |
| Crayfish | Houdini | ImageMagick |
| Crayfish | Hypercube | tesseract/pdftotext |
| Crayfits | Crayfits | FITS |
Derivatives are created using an external queue¶
To send orders to the microservices, Islandora sends messages to an external queue, from which the microservices process the jobs as they are able. This is a robust system that can "operate at scale", i.e. can large handle batches of uploads without slowing down the repository.
The queue system used by Islandora is ActiveMQ, and the listeners are part of "Alpaca"
Derivative Swimlane Diagram¶
The following diagram shows the flow of derivative generation from start to finish. A user saves a Media in Drupal, which may trigger Drupal to emit a derivative event to a queue, which is read by Alpaca and sent to a microservice. The microservice gets the original file and makes a transformation, returning the derivative file to Alpaca, which sends it back to Drupal to become a Drupal Media.
