feat(api, robot-server): Add source key to the data_files database to support generated CSV files from the plate reader + other plate reader fixes.

[vegano1]

Overview

The Plate Reader generates absorbance readings that can be optionally saved as a CSV file on the robot with the read(export_filename='filename.csv') PAPI command. This generated data needs to be stored on the robot so the client can download them later on for post-processing or re-upload them to be used as RTP CSV for a different protocol. Unlike RTP CSV that are uploaded by a user from the client, generated readings are produced by the robot. Because of this, we need to keep track of the source these data files originate from, so we can limit their count independently of RTP CSVs. A typical absorbance protocol makes about 40 readings per run, so we could hit our maximum number of data files in one run. This pull request adds a source key to the data_files table to keep track of the origin of the data files, changes to support the new functionality, and other fixes relating to the plate reader.

Closes: PLAT-574 PLAT-575

Test Plan and Hands on Testing

• Make sure we cant generate more than the maximum number of CSV files
• Make sure that you cant upload more than the maximum number of user RTP CSVs
• Run a protocol that does NOT call close_lid before initialize and make sure we raise CannotPerformModuleAction
• Make sure we add a source column to the data_files table in the database in v7 migration.
• The following protocol should run successfully and you should be able to download the csv files from the run logs.

from typing import cast
from opentrons import protocol_api
from opentrons.protocol_api.module_contexts import AbsorbanceReaderContext

# metadata
metadata = {
    'protocolName': 'Absorbance Reader Multi read/csv test',
    'author': 'Platform Expansion',
}

requirements = {
    "robotType": "Flex",
    "apiLevel": "2.21",
}

# protocol run function
def run(protocol: protocol_api.ProtocolContext):
    mod = cast(AbsorbanceReaderContext, protocol.load_module("absorbanceReaderV1", "C3"))
    plate = protocol.load_labware("nest_96_wellplate_200ul_flat", "C2")
    tiprack_1000 = protocol.load_labware(load_name='opentrons_flex_96_tiprack_1000ul', location="B2")
    trash_labware = protocol.load_trash_bin("B3")
    instrument = protocol.load_instrument("flex_8channel_1000", "left", tip_racks=[tiprack_1000])
    instrument.trash_container = trash_labware

    # pick up tip and perform action
    instrument.pick_up_tip(tiprack_1000.wells_by_name()['A1'])
    instrument.aspirate(100, plate.wells_by_name()['A1'])
    instrument.dispense(100, plate.wells_by_name()['B1'])
    instrument.return_tip()
   
    # Initialize to a single wavelength with reference wavelength
    # Issue: Make sure there is no labware here or youll get an error
    mod.close_lid()
    mod.initialize('single', [600], 450)

    # NOTE: CANNOT INITIALIZE WITH THE LID OPEN

    # Remove the Plate Reader lid using the Gripper.
    mod.open_lid()
    protocol.move_labware(plate, mod, use_gripper=True)
    mod.close_lid()

    # Take a reading and show the resulting absorbance values.
    # Issue: cant read before you initialize or you an get an error
    result = mod.read()
    msg = f"single: {result}"
    protocol.comment(msg=msg)
    protocol.pause(msg=msg)

    # Initialize to multiple wavelengths
    protocol.pause(msg="Perform Multi Read")
    mod.open_lid()
    protocol.move_labware(plate, "C2", use_gripper=True)
    mod.close_lid()
    mod.initialize('multi', [450, 570, 600])

    # Open the lid and move the labware into the reader
    mod.open_lid()
    protocol.move_labware(plate, mod, use_gripper=True)

    # pick up tip and perform action on labware inside plate reader
    instrument.pick_up_tip(tiprack_1000.wells_by_name()['A1'])
    instrument.aspirate(100, plate.wells_by_name()['A1'])
    instrument.dispense(100, plate.wells_by_name()['B1'])
    instrument.return_tip()

    mod.close_lid()

    # Take reading
    result = mod.read()
    msg = f"multi: {result}"
    protocol.comment(msg=msg)
    protocol.pause(msg=msg)

    # Take a reading and save to csv
    protocol.pause(msg="Perform Read and Save to CSV")
    result = mod.read(export_filename="plate_reader_csv.csv")
    msg = f"csv: {result}"
    protocol.pause(msg=msg)

    # Place the Plate Reader lid back on using the Gripper.
    mod.open_lid()
    protocol.move_labware(plate, "C2", use_gripper=True)
    mod.close_lid()

Changelog

• Raise CannotPerformModuleAction analysis error if initialize is called before close_lid
• Add a source column to the data_files in the database to keep track of the origin of the data file.
• Use the source key to determine the number of generated files stored on the bot to prevent file creation.
• Increase the maximum number of generated CSV files to 400 to account for the average case of number of reads a plate reader could make per protocol run.

Review requests

Risk assessment

[CaseyBatten]

Looks solid overall to me, the migrations work seems to have sensible coverage as well. I had a question/proposal down below regarding the current scope of uploaded/generated sources, but other than that should be good.

[CaseyBatten]

This should be fine, but just as an aside we're using this same limit within an actual protocol run as well so someone could theoretically do 400 file writes during a single protocol. Not necessarily a problem, just means they'll use up all their write limit in one go.

[CaseyBatten]

Very nice, makes sense to separate our deletion methodology like this.

[CaseyBatten]

Generated should be fine in general, but to look forward do we want to categorize these as something like "generated_csv" or more specifically "plate_reader_csv"? Would there be a reason in the future to want to tell the difference between types of generated/protocol output files via the file source field?

[SyntaxColoring]

I think that is a possibility. I wouldn't mind either way. If you keep it like this for v8.2, changing it to the other way is an easy and cheap migration to do later, if we need to.

[SyntaxColoring]

Reviewed the SQL bits, which all look good to me. Thanks!

I'll say though that taking a few steps back, I don't see a fundamental reason why generated and uploaded files couldn't be in one pool with a large upper limit, like in the hundreds or thousands. And I think we should try to steer ourselves towards that direction—one good fast persistence strategy—instead of adding increasingly granular "kinds" of things (quick transfer runs+protocols instead of normal ones, generated files instead of uploaded ones, ...) for the sake of detouring around existing limits.

[SyntaxColoring]

I think that is a possibility. I wouldn't mind either way. If you keep it like this for v8.2, changing it to the other way is an easy and cheap migration to do later, if we need to.

[vegano1]

Reviewed the SQL bits, which all look good to me. Thanks!

I'll say though that taking a few steps back, I don't see a fundamental reason why generated and uploaded files couldn't be in one pool with a large upper limit, like in the hundreds or thousands. And I think we should try to steer ourselves towards that direction—one good fast persistence strategy—instead of adding increasingly granular "kinds" of things (quick transfer runs+protocols instead of normal ones, generated files instead of uploaded ones, ...) for the sake of detouring around existing limits.

While I do agree with not adding different kinds of objects, In this case, the source key is used for

1. Limiting the count of generated files without impacting uploaded files
2. We don't want to auto-delete generated files like we do when a user uploads an RTP CSV file.
   1. Since you can download this data from the runs screen, we don't want to delete it once we reach the limit
3. Not showing generated CSV files when picking RTP CSV files a user has uploaded. We want to keep that distinction because we don't want to flood the user with 100s of generated files when they might be looking for something specific.