Often, we are not interested in individual samples or results on their own but rather a collection of samples and results executed against one or more pipeline versions. Or, we have a need to quickly and consistently execute a pipeline version on a large number of samples. In these cases, the Batch Analysis tool can be used for simple and traceable execution.
Creating a Batch Analysis
To create a batch analysis, simply go to the Batch Analysis tool and click + Add in the top right corner. Enter a name for the batch and choose one or more pipeline versions to use for execution. You may optionally add samples at this time from the Sources dropdown, or add them after creation.
Adding Samples to a Batch Analysis
There are two options to add samples to a batch analysis after creation.
Add Individual Samples: To add individual samples, click Add Sources from the Quick Actions menu. In the modal that pops up, choose your desired sources from the multiselect dropdown. Click Add Sources to complete.
Add Samples by Tag: To add samples based on their tags, click Add Sources by Tag from the Quick Actions menu. In the modal that pops up, choose your desired combination of ta
gs to require and exclude. As a sanity check, click Check Sources to find out how many items this brings back. If you're satisfied with the filtering, click Add Sources to complete.
Executing Pipelines in a Batch Analysis
The batch analysis overview page shows you all samples and pipelines included in your analysis, and the execution status for each combination. Typically, you will run these as a batch for convenience, but in some cases you might want to run combinations individually (for example, to test a pipeline on a single sample before kicking it off for hundreds).
Executing a Pipeline on All Batched Samples
At the top of the column corresponding to each pipeline in the batch analysis, you should see a "play" button. Clicking this button will kick off any un-started analyses for that pipeline.
Executing on Individual Sample/Pipeline Combinations
For any combinations that have not yet been kicked off, you can click the "play" button in the corresponding column to kick it off individually.
Tracking Progress in a Batch Analysis
To monitor progress of a batch analysis, first ensure that you are a "watcher" of the analysis. If you created the analysis, you will automatically have been assigned as a watcher. Otherwise, click the eyeball icon at the top right of the batch analysis overview page to watch. The batch analysis will now show up in your batch analyses widget, accessible from the home screen.
The batch analysis overview page allows you to track the progress of all executions in the batch. At the top of the page, the aggregate Summary is reported with status counts for each pipeline version. The Executions table shows the individual execution statuses for each sample-pipeline combination. Click on a combination's status to see more execution details or drill into results.
Manual-Upload Batch Analyses
In some cases, you may wish to batch-upload results from analyses run offline. In this case, you can create a batch analysis and target a Manual Upload executor. The difference for manual-upload batch analyses is that you will be linking the results yourself. There are two ways to do this:
Individual Results Upload - This follows the standard process for manual execution results upload. To get to the upload screen, open the Execution page for each combination (by clicking the status link in the Executions table). You will then find a button to manually upload results, as well as a copy-able cloud command if you prefer to do it from your cloud console. When you are done uploading, mark the execution as Complete, and Miqa will begin auto-processing the results.
Batch Results Upload (Advanced) - For advanced users with large batches of offline analysis, you may prefer to upload your results programmatically. To do so, you'll execute your copy command for each sample, linking that sample's results to the manual-execution output folder. A list of batch analysis samples and their corresponding execution output folders can be exported in CSV form by selecting Output Directory Listing from the Quick Actions menu. When you have finished copying over all relevant files, click Mark Manuals Complete from the Quick Actions menu to notify Miqa that these uploads are ready to be processed.