Supplementary Files are files that can be accessed by your pipelines at execution time and managed directly through Miqa. The best use case for supplementary files are small, human-readable (e.g. plaintext, JSON, YAML, python, bash, etc.) files that you use to configure or support your executions. Large and binary files are better suited to separate storage. Contact your Magna Labs support team for detailed recommendations for your use case.

Like Component Execution Templates, supplementary files can also (optionally) inherit runtime variables from the Executable Wrapper. As with templates, the variables should be designated with $-prefixed variable names corresponding to wrapper variable names, for example $BAM_PATH.

To properly replace these variables at runtime, you will need to use the variable replacement/environment preparation step in your workflow. This step should occur before the actual component execution steps and will reference a replacement-script such as "replace.py". If you do not already have this step in your workflow and would like to add one, reach out to your Magna Labs support team.

To access, add, and modify supplementary files, go to the Supplementary Files Manager page. You can reach this from Pipeline > Resources > Supplementary Files.

Click the + Add button from the Supplementary Files Manager to add a new supplementary file. You will be prompted with a form showing basic parameters for creating a supplementary file: the name, file type, and modification setting.

Name

The file name can be modified at any time. Please note that this corresponds to the name you see in the portal, and not necessarily the cloud-stored file key. The cloud-stored file key is generated only once, when the file is first added. If you need to change the underlying cloud file name, consider creating a new or duplicate file, or contact support.

File Type

The file type can be changed at any time and dictates the file editor view and download extension. After changing the file type, you must first Save the form for the changes to take effect.

File Contents

In-browser editing

If using the standard modification setting, you can edit your file right in the browser using the file editor, and then hit Save to submit the changes. When entering Modify In-Browser mode, make sure to have the Modify Locally box unchecked in the Edit Supplementary File modal and when creating the new Supplementary File. If you wish to upload a locally-modified file but have created a file in-browser, you will have to back out and create a new Modify-Locally file.

Modify Locally

If the Modify Locally setting is checked, you will be able to download and upload files but not edit in the editor. You do not need to click Save when re-uploading; the files will automatically be uploaded and reloaded. When entering Modify Locally mode, make sure to have the Modify Locally box checked in the Edit Supplementary File modal and when creating the new Supplementary File. If you wish to edit a file in-browser but have selected to modify locally, you will have to back out and create a new In-Browser file.

When you have material modifications to make to a supplementary file, we recommend creating a separate file rather than modifying the original, so that the original can still be used in executions of older workflows where the new version may not be compatible.

To duplicate a supplementary file, go to the Edit Supplementary File view and use the Duplicate button in the Quick Actions dropdown menu.

You will be taken to a form where you can specify the name of the new file before going through with the duplication. All other file settings will be copied from the original file.

When you click Save, you will be taken to the Edit Supplementary File page for the newly duplicated file. You can see that all settings have carried over, as well as the file contents*. You may now make any modifications to the new file and save it. Don't forget to use the new key in any forward-looking component templates! (see Using Supplementary Files below).

*Please note that for offline/"Modify Locally" files, the file contents themselves will not be copied. You must upload a new file to be used at this location.

Supplementary files can be referenced directly from your execution template via their cloud key. The cloud key can be retrieved from the Edit Supplementary File page by hovering over the eye icon to the right of Show Key or clicking on the copy button to copy it to your clipboard. This key can then be pasted into your component execution template. Make sure to properly reference the cloud path (i.e. gs://{KEY} or s3://{KEY} for your organization's environment).

If you have modifications to a supplementary file that do not impact its backwards/forwards compatibility (e.g. fixing a breaking typo), you may edit the file and there is no need to modify any references to the supplementary file as your modified version will now be present at the original cloud location. Exercise caution when modifying supplementary files that are already used by one or more pipelines. It is recommended that you use this approach only for newly referenced files (e.g. new file failed it's first execution because of a typo).

If you are modifying a file in use by one or more pipelines, and especially if these modifications may impact its backwards/forwards compatibility, we recommend creating a new file (or duplicating the original and making your modifications), and then creating a new version of the Component Execution Template pointing to the new key. This allows your template and supplementary files to move in lockstep, enabling easy backwards compatibility and ensuring complete reproducibility of analyses.