Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JBrowse2 tracks deconvolution #129

Open
nekrut opened this issue Oct 10, 2024 · 18 comments
Open

JBrowse2 tracks deconvolution #129

nekrut opened this issue Oct 10, 2024 · 18 comments
Assignees
@scottcain

Comments

@nekrut
Copy link
Contributor

nekrut commented Oct 10, 2024

  • VEuPathDB had JBrowse instances set up so that each config and tracks are done dynamically (not typical); REST service sits in front of database to write config files and extract data dynamically to display in JBrowse
  • For sequence and genes, .gff and .fasta; not accessed
  • There are many many tracks for plasmodium genome
  • PlasmoDB flat files are massive
  • Can see github repo where we could extract information from other data tracks and database features, can reverse engineer from here
@nekrut nekrut converted this from a draft issue Oct 10, 2024
@nekrut
Copy link
Contributor Author

nekrut commented Oct 10, 2024 •
edited
Loading

Since VEuPathDb is accessible now, can we do this via some kind of crawling @scottcain ?

@nekrut
Copy link
Contributor Author

nekrut commented Oct 10, 2024

Contingent on our ability to query the db

@maximilianh
Copy link

maximilianh commented Oct 10, 2024 via email

@nekrut
Copy link
Contributor Author

nekrut commented Oct 10, 2024

Can't this be done via the jbrowser API endpoint, the one that jbrowse uses? Veupathdb staff told us how to do this...
…
On Thu, Oct 10, 2024 at 6:14 PM Anton Nekrutenko @.> wrote: Contingent on our ability to query the db — Reply to this email directly, view it on GitHub <#129 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TP2SNIMJXYHEJGCHSTZ22RWDAVCNFSM6AAAAABPXATNM6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBVGUZTKMJQGA . You are receiving this because you are subscribed to this thread.Message ID: @.>

We just discussed this with @scottcain

@maximilianh
Copy link

Was the conclusion that it's possible using the API? (sorry I had a meeting at exactly the same time and duration as the BRC meeting today)

@scottcain
Copy link
Collaborator

@maximilianh it depends on what we want to do. Since these tracks' data are now static from VEuPathDB (that is, they aren't being actively curated, which would be a reason you might want to run the tracks from a database query), using the database to serve up JBrowse data is (in my opinion) a little obnoxious, when the alternative is something like tabix indexed GFF, BigBed or BigWig (along with associated metadata in all cases). Shifting to static files sitting in a bucket or web server somewhere would make implementing JBrowse 2 a lot easier (and presumably other genome browsers 😉), and is better for the community, since it makes getting a whole dataset for a given track a lot easier.

The next question is, could we write a spider that crawled all of the JBrowse instances at VEuPathDB and extracts all of the data? Well, yes, I suppose we could, but it would be kind of an unfriendly thing to do. Rather, I think a reasonable approach is wait for a new hire in Sergei's group who knows this database pretty well and can (presumably) help us extract the track data and metadata from our instance of the database.

@maximilianh
Copy link

maximilianh commented Oct 10, 2024 via email

@maximilianh
Copy link

maximilianh commented Oct 10, 2024 via email

@d-callan
Copy link
Collaborator

so that was fun lol.. first thoughts as i read this:

  1. im not completely sure a spider is too unfriendly, and might be handy
  2. wrt using the db vs flat files, the db predated jbrowse. it already existed w the data needed and flat files didnt. it also meant one source of truth for all parts of the site rather than keeping the db and flat files in sync. if they manage sustainability, they also want to move to jbrowse2 and all flat files if they have the time and resources
  3. id be generally curious what all tracks you wanted to reproduce, and their priority.
  4. id also be curious where else you thought you might want to use some of these data. the db is a pretty purpose built thing. its job is to run a veupath genomics site. as much as there is a lot of great stuff in there, theres also some notable stuff missing, bc what wasnt intended to be used in a search or a plot wasnt loaded. that might impact your general strategy for trying to digest veupath data, which might impact plans for this task? dunno. to give an ex though, for rnaseq experiments tpm was loaded but not counts. if you wanted to try to incorporate these data in a different context in a way that would allow something like deseq to be used on them, thatd be a problem.
  5. related to previous i suppose, i wonder what if anything besides the db you were given?

@scottcain
Copy link
Collaborator

@d-callan these are solid questions. Here are my thoughts:

  1. im not completely sure a spider is too unfriendly, and might be handy

Perhaps, but I still think it would (likely) be cleaner to extract, format and config data coming directly from the database. And of course, such a spider doesn't actually exist except in my head.

  1. wrt using the db vs flat files, the db predated jbrowse. it already existed w the data needed and flat files didnt. it also meant one source of truth for all parts of the site rather than keeping the db and flat files in sync. if they manage sustainability, they also want to move to jbrowse2 and all flat files if they have the time and resources

Also fair--I can understand that as a design decision; on the other hand, I very much like having JBrowse backed by flatfiles, since it makes it easy for users who want to get a genome's worth of a particular data set much easier.

  1. id be generally curious what all tracks you wanted to reproduce, and their priority.

My philosophy can generally be stated as "if the data exists, let's get it in JBrowse" This can lead to a discoverability issue for assemblies with lots of data, but that's not a terrible problem to have, and tools are being developed to address it.

  1. id also be curious where else you thought you might want to use some of these data. the db is a pretty purpose built thing. its job is to run a veupath genomics site. as much as there is a lot of great stuff in there, theres also some notable stuff missing, bc what wasnt intended to be used in a search or a plot wasnt loaded. that might impact your general strategy for trying to digest veupath data, which might impact plans for this task? dunno. to give an ex though, for rnaseq experiments tpm was loaded but not counts. if you wanted to try to incorporate these data in a different context in a way that would allow something like deseq to be used on them, thatd be a problem.

It is unclear to me if there is any real use for any of the other data in the database--when we got it, we were thinking VEuPathDB was going away for good. With it still existing, we can make better use of the website than trying to build something ourselves (in my opinion, that is).

  1. related to previous i suppose, i wonder what if anything besides the db you were given?

I am reasonably sure that we got a bunch of binary files like bigwigs and bams, but I don't know that for sure (but I hope we did!) @jdavcs might know more

@scottcain
Copy link
Collaborator

Although, it just occurred to me that, since the JBrowse instance at VEuPathDB continues to exist, we could potentially proxy (to get around CORS issues) the paths where the data extraction tools are and just use their data directly. I will try experimenting with that today.

@jdavcs
Copy link
Member

jdavcs commented Nov 14, 2024

@scottcain @d-callan We have roughly 11TB of flat files representing 13 DBs. I don't know what exactly we have, but here's a tiny sample:

Pyoeliiyoelii17X
├── bam
├── bigwig
├── blast
├── blat
├── config
├── fasta
├── gff
├── highSpeedSnpSearch
├── longReadRNASeq
├── motif
└── nrProteinsToGenomeAlign

@scottcain
Copy link
Collaborator

More thoughts for @d-callan : what would be really cool would be a tool that, given the assembly we want it for, would extract the information needed from the database for what tracks are available and construct a config. There are a few details hidden in that one sentence, like

  • when the track is derived from a binary file like bigwig or bam, figuring out where that is in the file system John referred to above would be helpful (I'm guessing not too difficult)
  • when the track is derived from data in the database, it will have to be spit out into a flatfile that makes sense, and then probably be post processed (bgzipped and tabix indexed gff, for example)
  • generating JBrowse 2 configs would be helpful, rather than JBrowse 1 (not hard, just different)
  • generating JB2 configs also assumes that the assembly's fasta file is indexed in some way (like bgzip and faidx); that isn't your problem but something that will need to be addressed.

@d-callan
Copy link
Collaborator

so if were serious about this.. id say that tool already exists. its at/ sort of is veupath. which means for us that tool is a spider.. no?

ex: https://plasmodb.org/plasmo/service/jbrowse/tracks/pvivP01/trackList.json and follow it round.

@maximilianh
Copy link

maximilianh commented Nov 20, 2024 via email

@d-callan
Copy link
Collaborator

d-callan commented Dec 9, 2024

so some questions relating to this:

  1. How can we provide context for all these data? the dataset and sample display names from veupath are unlikely to be inherently meaningful to users.
  2. For the experimental data, do we think ppl would want to play w it in galaxy? say do something like find differentially expressed genes in an rnaseq experiment, say? and if so, how can we facilitate that?

@maximilianh
Copy link

maximilianh commented Dec 9, 2024 via email

@d-callan
Copy link
Collaborator

d-callan commented Dec 9, 2024

  1. Idk tbh, links to veupath dataset record pages? Or if we tried to get metadata what could we do w it? What would be worth trying to get?
  2. This is a thing I'd like to play w..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@scottcain scottcain

Labels
None yet
Projects
BRC development tasks
Status: In Progress
Milestone
No milestone
Development

No branches or pull requests
5 participants
@scottcain @maximilianh @nekrut @jdavcs @d-callan