2021-08-10 weekly planning

[kgodey]

Discussion topics

• Retrospective: Summary of last week's work.
• Active Work: Work planned for the next week.
• Updates: General updates everyone should be aware of.
• Questions: Questions for the team to answer.

About

Each Tuesday, we start a discussion about Mathesar's progress. Everyone is welcome to add a new topic to discuss or to comment on existing topics.

[kgodey]

Retrospective

This post lists issue updates from 2021-08-03 until now by milestone.

03. Tables from File Import

• Issues closed:
  • [@mathemancer] API updates for confirm data step in CSV file import. #201
• Issues started, not yet finished:
  • [@pavish] File import preview page #295

04. Working with Schemas

• Issues closed:
  • [@pavish] Implement new top navigation and layout #362
  • [@pavish] Switch to selected schema in file import view #316
  • [@dhruvkb] Implement schema list page #171
• New issues created:
  • Schema list page interactivity #518, created by @pavish

05. Editable Tables

• Issues started, not yet finished:
  • [@pavish] Implement record editing #195
• New issues created:
  • Switch open tables when schema is changed #524, created by @pavish

06. Working with Tables

• Issues closed:
  • [@eito-fis] Table API should support creating tables without names #449
  • [@eito-fis] Create canonical ordering for tables and return it by default #397
  • [@andyharley-cci] Table creation returns 500 if there's a table with the same name #364
  • [@kgodey] CRUD API for table constraints #326
  • [@ghislaineguerin] Design for editing column properties and table constraints #324
  • [@eito-fis] API to upload file from URL #203
• Issues started, not yet finished:
  • [@eito-fis] Extend column API to support duplicating columns #452
  • [@eito-fis] DB and API work for adding a default value to columns #494
  • [@pegnott] Implement adding a new column #495
• New issues created:
  • Frontend work for adding multi-column unique constraints #531, created by @kgodey

07. Initial Data Types

• Issues closed:
  • [@mathemancer] Handle DOUBLE PRECISION data type in the backend. #383
  • [@mathemancer] Handle REAL data type in the backend. #382
  • [@mathemancer] Handle DECIMAL data type in the backend. #381
  • [@mathemancer] Handle NUMERIC data type in the backend. #377
  • [@ghislaineguerin] Design for Number data types #238
  • [@ghislaineguerin] Design for text data type #234
• Issues started, not yet finished:
  • [@ghislaineguerin] Design for boolean data type #237
  • [@ghislaineguerin] Design for URI data type #236
  • [@ghislaineguerin] Design for email data type #235
  • [@powellc] Handle BOOLEAN data type in the backend #387
  • [@mathemancer] Handle SMALLINT data type in the backend. #378
  • [@mathemancer] Handle BIGINT data type in the backend. #380
  • [@mathemancer] Handle INTEGER data type in the backend. #379
  • [@kgodey] Store friendly to DB type mapping in the backend #367
  • [@kgodey] Extend the v0/databases/'s types list to show both Mathesar and DB type information. #368
  • [@kgodey] Ensure that the API only accepts DB types as input everywhere we use types. #369
  • [@ghislaineguerin] Design for date & time data types #240
  • [@ghislaineguerin] Design for duration data type #241
  • [@ghislaineguerin] Design for money data type #239
• New issues created:
  • Design for data types that are not "Mathesar types" #537, created by @kgodey

2021-08 improvements

• Issues closed:
  • [@andyharley-cci] Staging deploys failing at host verification step #534
  • [@kgodey] Separate GH actions for repo branches and forks #479
• Issues started, not yet finished:
  • [@andyharley-cci] Handle non-unicode encoded datafiles #421
  • [@mr-gabe49] Select component keyboard accessibility #483
  • [@pavish] Serve built static files in production #358
  • [@andyharley-cci] Set up staging environment for Mathesar #225
  • [@Dhruvi16] Create button component in the component library. #322
• New issues created:
  • Staging deploys failing at host verification step #534, created by @andyharley-cci
  • Storybook does not resolve paths from TS config #528, created by @dhruvkb

09. Data Modeling

• New issues created:
  • Create design and backend issues for Data Modeling milestone #538, created by @kgodey

10. Multiple Database Support

• Issues closed:
  • [@pavish] Switch frontend code to the database API #312
• Issues started, not yet finished:
  • [@aaditkamat] Backend support for tables without single-column primary keys #395

[kgodey]

Active Work

This post lists active work based on the Active GitHub project. It also includes a current summary of milestones involved in active work.

[2021-08-11] 03. Tables from File Import

• 27 issues already closed.
• Issues currently in progress:
  • [@pavish] File import preview page #295
• Issues not yet started:
  • [@pavish] Import status indicators #296

[2021-08-12] 04. Working with Schemas

• 13 issues already closed.
• Issues not yet started:
  • [@pavish] Schema list page interactivity #518

[2021-08-16] 05. Editable Tables

• 6 issues already closed.
• Issues currently in progress:
  • [@pavish] Implement record editing #195
• Issues not yet started:
  • [@pavish] Implement adding new records #197
  • [@pavish] Switch open tables when schema is changed #524

[2021-08-23] 06. Working with Tables

• 14 issues already closed.
• Issues currently in progress:
  • [@eito-fis] Extend column API to support duplicating columns #452
  • [@eito-fis] DB and API work for adding a default value to columns #494
  • [@pegnott] Implement adding a new column #495
• Issues not yet started:
  • [@eito-fis] Add filter to show records with duplicate values in a set of columns #501
  • [@eito-fis] DB function to get rows with duplicate values #515
• Issues remaining in backlog:
  • 14 frontend issues

[2021-08-30] 07. Initial Data Types

• 8 issues already closed.
• Issues currently in progress:
  • [@ghislaineguerin] Design for boolean data type #237
  • [@ghislaineguerin] Design for URI data type #236
  • [@ghislaineguerin] Design for email data type #235
  • [@powellc] Handle BOOLEAN data type in the backend #387
  • [@mathemancer] Handle SMALLINT data type in the backend. #378
  • [@mathemancer] Handle BIGINT data type in the backend. #380
  • [@mathemancer] Handle INTEGER data type in the backend. #379
  • [@kgodey] Store friendly to DB type mapping in the backend #367
  • [@kgodey] Extend the v0/databases/'s types list to show both Mathesar and DB type information. #368
  • [@kgodey] Ensure that the API only accepts DB types as input everywhere we use types. #369
  • [@ghislaineguerin] Design for date & time data types #240
  • [@ghislaineguerin] Design for duration data type #241
  • [@ghislaineguerin] Design for money data type #239
• Issues not yet started:
  • [@powellc] Implement filtering options for Boolean types #388
  • [@powellc] Implementation for boolean data type #247
  • [@mathemancer] Backend work for custom MATHESAR_TYPES.PERCENTAGE type. #434
  • [@eito-fis] Spec for implementing display options for Columns #392
  • [@eito-fis] Handle TIMESTAMP data type in the backend #424
  • [@eito-fis] Handle DATE data type in the backend #425
  • [@eito-fis] Handle TIME data type in the backend #426
  • [@eito-fis] Implement filtering options for date & time types #427
  • [@eito-fis] Implement grouping options for date & time types #429
  • [@rootxnet] Handle VARCHAR data type in the backend. #404
  • [@rootxnet] Handle TEXT data type in the backend. #405
  • [@rootxnet] Handle CHAR data type in the backend. #407
  • [@rootxnet] Implement filtering options for Text types. #406
  • [@rootxnet] Implement grouping for Text types. #408
  • [@kgodey] Backend work for Mathesar email custom type. #409
  • [@kgodey] Implement filtering options for the email type #410
  • [@kgodey] Implement grouping options for the email type #411
  • [@ghislaineguerin] Design for data types that are not "Mathesar types" #537
  • [@mathemancer] Backend work for Mathesar money custom type  #417
  • [@mathemancer] Create URI custom type and handle it in the backend. #412
  • [@powellc] Implement range based grouping for numeric types. #384
  • [@powellc] Implement filtering options for Number types. #385
• Issues remaining in backlog:
  • 20 backend and database issues
  • 8 frontend issues

[2021-08-30] 08. Working with Views

• 4 issues already closed.
• Issues not yet started:
  • [@ghislaineguerin] Design for usage of Foreign Key constraints #243
  • [@ghislaineguerin] Support for deleting a database view #454
  • [@ghislaineguerin] Support for listing database views #455
  • [@ghislaineguerin] Support for editing records within a view #456
  • [@ghislaineguerin] Support for opening and interacting with a View #466
• Issues remaining in backlog:
  • 8 backend and database issues
  • 1 frontend issues
  • 1 design issues
  • 1 other issues

[2021-08-31] 2021-08 improvements

• 6 issues already closed.
• Issues currently in progress:
  • [@andyharley-cci] Handle non-unicode encoded datafiles #421
  • [@mr-gabe49] Select component keyboard accessibility #483
  • [@pavish] Serve built static files in production #358
  • [@andyharley-cci] Set up staging environment for Mathesar #225
  • [@Dhruvi16] Create button component in the component library. #322
• Issues not yet started:
  • [@mathemancer] Schema sqlalchemy engine caching is per-instance #280
• Issues remaining in backlog:
  • 3 backend and database issues
  • 10 frontend issues
  • 6 design issues
  • 2 infrastructure issues

[2021-09-02] 09. Data Modeling

• 3 issues already closed.
• Issues not yet started:
  • [@kgodey] Create design and backend issues for Data Modeling milestone #538
• Issues remaining in backlog:
  • 4 backend and database issues
  • 3 frontend issues
  • 2 design issues

[2021-09-06] 10. Multiple Database Support

• 7 issues already closed.
• Issues currently in progress:
  • [@aaditkamat] Backend support for tables without single-column primary keys #395
• Issues remaining in backlog:
  • 3 backend and database issues
  • 3 frontend issues
  • 2 design issues

[kgodey]

Updates

• Milestones 03. Tables from File Import and 04. Working with Schemas have switched order to account for implementation time, but both are still due this week.

• The Mathesar update presentation to the CCI research group went well. I recorded the Zoom call and I'm considering uploading the video somewhere public (like YouTube) so that it can be a resource to future contributors or other interested parties. What do you all think?

  • @aaditkamat @powellc @dhruvkb @mr-gabe49 @obulat thanks for attending!

[ghislaineguerin]

@kgodey Good idea, I think uploading it makes sense.

[mathemancer]

I think it would be great to have as well.

[kgodey]

Questions

@mathemancer, @eito-fis, @ghislaineguerin, @pavish: Your responses are required. Others are also welcome to respond.

Retrospective

• What went well this week?
• What could have gone better this week?
• Are there any improvements to our process you'd like to suggest?

Active Work

• Does your workload for this week look okay?
• We have three milestones due in the next week. Are there any blockers to meeting our upcoming deadlines? (mainly a question for @pavish)

Discussion

• Are the auto-generated active work and retrospective sections useful? Are there any format changes you'd like to suggest?
• Anything else you'd like to discuss?

[eito-fis]

What went well this week?

Got to merge some PRs that had been open for a while. Also feeling much more comfortable with sqlalchemy queries and postgres system tables.

What could have gone better this week?

Feeling my inexperience with the types side of things a bit, would like to find some time to dig into how sqlalchemy and postgres handles types more thoroughly.

Are there any improvements to our process you'd like to suggest?

N / A

Does your workload for this week look okay?

Yep, seems fine.

We have three milestones due in the next week. Are there any blockers to meeting our upcoming deadlines?

N / A

Are the auto-generated active work and retrospective sections useful? Are there any format changes you'd like to suggest?

The auto-generated section are awesome, no changes to suggest.

Anything else you'd like to discuss?

N / A

[ghislaineguerin]

What went well this week?

Reviews have been helpful and it's good to have them in parallel with new designs so that they inform current decisions.

What could have gone better this week?

I think I took a bit longer with the last data types as we had some UI changes that had to be made across all prototypes. A consolidated prototype with components would be best but also slows down the design initially.

Are there any improvements to our process you'd like to suggest?

Not for now. I will try to consolidate all the work with 'Views' into a single prototype and see how that goes.

Does your workload for this week look okay?

It does.

We have three milestones due in the next week. Are there any blockers to meeting our upcoming deadlines? (mainly a question for @pavish)

No.

Are the auto-generated active work and retrospective sections useful? Are there any format changes you'd like to suggest?

Yes, I think they are useful. Not for now.

Anything else you'd like to discuss?

Not for now.

[mathemancer]

What went well this week?

I enjoyed knocking out a few issues. I also think the presentation went well. I'm pleasantly surprised that the way of implementing type support seems to be coming together so that most type implementations should be quicker and quicker moving forward.

What could have gone better this week?

I kept getting further away from the master branch than I'd prefer, since I was usually working on different branches that depended on each other. I think some of the re-organization I did w.r.t. testing and type implementations should alleviate the need to make dependent branches moving forward.

Are there any improvements to our process you'd like to suggest?

I think we already made the main improvement I wanted this week, having to do with how to deal with failing npm audit results.

Does your workload for this week look okay?

Yes, with the caveat that I may be out for parental leave at some unpredictable moment soon.

We have three milestones due in the next week. Are there any blockers to meeting our upcoming deadlines?

Nothing I see.

Are the auto-generated active work and retrospective sections useful? Are there any format changes you'd like to suggest?

I can't decide whether putting the retro and active work together would be better. I kept scrolling back and forth to connect what was TODO with what was DONE to make sure the union of those two covered everything I could think of. I.e., it would be nice to have some overall view of an entire milestone. I suppose I can just click away for that, though, and it might be overwhelming here.

Anything else you'd like to discuss?

Nope!

[pavish]

What went well this week?

I'm glad we decided on the data loading approach. The presentation went well. The structural changes for db + schema selection is done, this stabilizes the schema store.

What could have gone better this week?

I had a number of interviews during the week. We also had the presentation meetings, and data loading discussions. This lead to less development time. It would have been better if I had factored in all of this before the start of the week, so that the milestone deadlines could have been set accordingly.

Are there any improvements to our process you'd like to suggest?

I think we need to improve judgement of the amount of frontend work, while setting deadlines.

Does your workload for this week look okay?
We have three milestones due in the next week. Are there any blockers to meeting our upcoming deadlines? (mainly a question for @pavish)

File import preview page and indicators, involve a bit of thought. The actual code is less, but we need to factor in a number of scenarios.
Schema listing page interactivity has a day's work.

I am hoping to complete milestone 03. Tables from File import and 04. Working with schemas by 16th (Monday).

However, there are 3 issues in Milestone 05. Editable tables have a number of changes, each of them would take a couple days atleast (they may lead to creation of further issues which would only become evident during development). We will have to extend the deadline for this.

Are the auto-generated active work and retrospective sections useful? Are there any format changes you'd like to suggest?

I find it useful. No suggestions.

Anything else you'd like to discuss?

None.

[kgodey]

@mathemancer I could do a single summary that included both retrospective and active work. There's some repeated data between both. I might try that if I have some time in the next couple of weeks.