2021-08-31 weekly planning

[kgodey]

Discussion topics

• Updates: General updates everyone should be aware of.
• Questions: Questions for the team to answer.
• Active Work: Work planned for the next week.
• Retrospective: Summary of last week's work.

About

Each Tuesday, we start a discussion about Mathesar's progress. Everyone is welcome to add a new topic to discuss or to comment on existing topics.

This post covers two weeks since we didn't do a discussion last week.

[kgodey]

Updates

Progress

We completed 3 milestones in the last 2 weeks!

• 03. Working with Schemas - 14 issues
• 04. Tables from File Import - 33 issues
• 2021-08 improvements - 23 issues

We've also now completed over 200 issues in total.

Roadmap Updates

Based on progress so far, we need to cut some features from our plan to ensure that we release a high-quality alpha version of Mathesar by the end of 2021. Our roadmap was fairly ambitious, so this is not a surprise. The Mathesar core team is meeting tomorrow to discuss the plan, but I made some initial changes to our roadmap to get us started. The plan is to focus on getting the experience of working with tabular data right for the alpha release before implementing complex new features like visualizations and real-time collaboration.

Please review the updated roadmap to see the latest state. We also added two new milestones to focus on stability - milestones 06 and 13. This will give us some decided time to address technical debt and fix issues with what we've built up to that point.

Core Team

• @eito-fis is done with his internship and is back to college. Thanks for all of your hard work, it was a pleasure to work with you!
• @mathemancer is back from parental leave.

Contributors

We have a couple of new contributors who had their first pull requests merged into Mathesar. Welcome to @asharonbaltazar, @linzhou-zhong, and @arvind-1. :)

Also, @mr-gabe49 had a couple more pull requests merged in, nice work!

[eito-fis]

Thanks, I learned a ton this summer! And I'll still be around to help with reviews and PRs 🙂

[kgodey]

Questions

@mathemancer, @powellc, @ghislaineguerin, @pavish: Your responses are required. Others are also welcome to respond.

Retrospective

• What went well over the past two weeks?
• What could have gone better over the past two weeks?
• Are there any improvements to our process you'd like to suggest?

Active Work

• Does your workload for this week look okay?
• We have one milestone due in the next week. Are there any blockers to meeting the deadline?

Discussion

Please respond to the Updates post directly if you'd like to discuss anything there. If you have any other topics to discuss, please post them as new replies to the parent post.

[kgodey]

What went well over the past two weeks?

With my work: I closed all the API bugs that Pavish opened, some of which involved major refactoring. I am more comfortable with the db code now.

With the project: We closed three milestones and we have more contributors. :). It's especially nice to get the tables from file import done, it was one of the first things we started work on.

What could have gone better over the past two weeks?

I can't think of anything specific. It took me longer to refactor the db code than I anticipated, but I'm not sure that could have been helped.

Are there any improvements to our process you'd like to suggest?

I think pausing feature work to do some refactoring is a good idea at this point, I'm glad we're doing it. But that's something we're already planning to do.

Does your workload for this week look okay?

I probably won't get everything done, I have 7 issues assigned to me, but they're mostly all refactoring related, so I think it makes sense to think about them together.

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

Not from me.

[mathemancer]

What went well over the past two weeks?

N/A

What could have gone better over the past two weeks?

N/A

Are there any improvements to our process you'd like to suggest?

Nothing I can think of at the moment

Does your workload for this week look okay?

I think so, though the mathesar MONEY type is a bit more complicated than I originally thought.

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

Nothing I can think of.

[powellc]

What went well over the past two weeks?

Finally getting comfortable with the full dev process, the code base, and especially the DB reflection. First PR merged and I'm hoping my development speed ramps up nicely from here.

What could have gone better over the past two weeks?

I need to be opening PRs and reaching out for comment on my solutions sooner. I think I probably could have been working faster the last week if I'd just opened the PR instead of sitting on the solution for a few days.

I also am still a little time limited due to personal obligations, so figuring out the best days and times to get focused work done can be improved, and then communicating that to the team.

Are there any improvements to our process you'd like to suggest?

Not at the moment.

Does your workload for this week look okay?

Yes. With a format taking shape for documenting filtering, and a good test frame for filtering types in the API, I should move faster through those tickets this week and be able to spend some time on Friday going deep on #248

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

Not from me.

[ghislaineguerin]

What went well over the past two weeks?

Global documents/prototypes seem to be working well to resolve the issues blocking the review process. I'm also getting a lot from doing quick design reviews w/ Kriti.

What could have gone better over the past two weeks?

Nothing specific. I think it went well in general.

Are there any improvements to our process you'd like to suggest?

Continue to do global documents at the end of a milestone. So, for example, once views are complete, have a prototype covering all prototypes for that milestone.

Does your workload for this week look okay?

It does.

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

Not that I'm aware of.

[pavish]

What went well over the past two weeks?

We're progressing on completing milestones. There were more external contributions for the frontend. Frontend work is moving in a steady pace.

What could have gone better over the past two weeks?

Didn't get to work much last week, due to health concerns. I think overall, things went well during the past two weeks.

Are there any improvements to our process you'd like to suggest?

Nothing at the moment.

Does your workload for this week look okay?

Yes, it looks good.

We have one milestone due in the next week. Are there any blockers to meeting the deadline?

None that I can think of.

[kgodey]

Active Work

This post lists work planned for this week (see also: Active). It also includes a current summary of milestones involved in this week's work.

[2021-09-07] 05. Editable Tables

• 8 issues already closed.
• Issues currently in progress:
  • [@pavish] Implement record editing #195
• Issues not yet started:
  • [@pavish] Implement adding new records #197

[2021-09-20] 06. 2021-09 Stability

• Issues currently in progress:
  • [@andyharley-cci] Set up staging environment for Mathesar #225
• Issues not yet started:
  • [@kgodey] Schema sqlalchemy engine caching is per-instance #280
  • [@kgodey] Reorganize code in db and set some standards for imports #582
  • [@kgodey] Set up logging in the backend code #583
  • [@kgodey] Refactor db code to work with existing connections #592
  • [@kgodey] Split up API and serializer code by object #510
• Issues remaining in backlog:
  • 3 backend and database issues
  • 1 frontend issues
  • 1 design issues

[2021-09-27] 07. Working with Tables

• 19 issues already closed.
• Issues currently in progress:
  • [@raduab] Frontend work for viewing and deleting table constraints #328
  • [@seancolsen] Implement adding a new empty table #444
  • [@dmos62] Implement showing and changing a column's type #498
  • [@pegnott] Implement adding a new column #495
• Issues not yet started:
  • [@Lucasmiguelmac] Implement setting a default value for a column #500
• Issues remaining in backlog:
  • 10 frontend issues

[2021-09-30] 2021-09 improvements

• 2 issues already closed.
• Issues currently in progress:
  • [@asharonbaltazar] CellHeader dropdown floats above the sidebar #594
  • [@arvind-1] Column API should support creating columns without names  #555
  • [@linzhou-zhong] Uncaught 404s in record viewsets and pagination classes #517
  • [@andyharley-cci] Handle non-unicode encoded datafiles #421
  • [@mr-gabe49] UpArrow does not work in the group section dropdown when data is grouped. #609
• Issues remaining in backlog:
  • 10 frontend issues
  • 3 infrastructure issues

[2021-10-04] 08. Initial Data Types

• 26 issues already closed.
• Issues currently in progress:
  • [@powellc] Handle BOOLEAN data type in the backend #387
  • [@mathemancer] Backend work for Mathesar money custom type  #417
  • [@eito-fis] Handle TIME data type in the backend #426
  • [@powellc] Implement filtering options for Boolean types #388
• Issues not yet started:
  • [@kgodey] Update implementation issues for Initial Data Types milestone #552
  • [@kgodey] Spec for implementing display options for Columns #392
  • [@rootxnet] Handle VARCHAR data type in the backend. #404
  • [@rootxnet] Handle TEXT data type in the backend. #405
  • [@rootxnet] Handle CHAR data type in the backend. #407
  • [@powellc] Implement range based grouping for numeric types. #384
  • [@powellc] Implement filtering options for Number types. #385
  • [@mathemancer] Create URI custom type and handle it in the backend. #412
  • [@mathemancer] Handle TIMESTAMP data type in the backend #424
  • [@mathemancer] Handle INTERVAL type in the backend #430
• Issues remaining in backlog:
  • 25 backend and database issues
  • 8 frontend issues

[2021-10-18] 09. Working with Views

• 4 issues already closed.
• Issues currently in progress:
  • [@ghislaineguerin] Support for opening and interacting with a View #466
  • [@ghislaineguerin] Support for deleting a database view #454
  • [@ghislaineguerin] Design for usage of Foreign Key constraints #243
• Issues not yet started:
  • [@ghislaineguerin] Support for listing database views #455
  • [@ghislaineguerin] Support for editing records within a view #456
  • [@ghislaineguerin] Design for many-to-many relationships between tables  #443
• Issues remaining in backlog:
  • 7 backend and database issues
  • 1 frontend issues
  • 1 other issues

[2021-11-01] 11. Multiple Database Support

• 7 issues already closed.
• Issues currently in progress:
  • [@aaditkamat] Backend support for tables without single-column primary keys #395
• Issues remaining in backlog:
  • 4 backend and database issues
  • 3 frontend issues
  • 2 design issues

[kgodey]

Retrospective

This post lists issue updates from 2021-08-17 until now by milestone.

03. Working with Schemas

• Issues closed:
  • [@pavish] Schema list page interactivity #518

04. Tables from File Import

• Issues closed:
  • [@pavish] Use suggested types in import preview step #568
  • [@kgodey] Tables PATCH endpoint does not update column properties #562
  • [@pavish] File import preview page #295

2021-08 improvements

• Issues closed:
  • [@kgodey] Table OID is not unique validation error, when creating a table using a data_file #595
  • [@kgodey] New setup fails during the install step #579
  • Error importing CSV file through the frontend #577
  • [@kgodey] Patch request for schema with same name results in 500 #576
  • [@linzhou-zhong] Double quotes in file headers break CSV import #572
  • [@kgodey] Patch on a table, with same name results in a 500 #566
  • [@kgodey] Call to columns patch endpoint with both name and type results in 500 #564
  • [@kgodey] Previews endpoint casts a column and returns 200, while column patch endpoint fails with 500 #563
  • [@eito-fis] Duplicate tables are present with the same oid in mathesar_table #559
  • [@pavish] Storybook does not resolve paths from TS config #528
  • [@kgodey] Serve built static files in production #358
  • [@Dhruvi16] Create Tables component for the components library. #334
  • [@Dhruvi16] Create collapsable headers component for the component library. #333
  • [@Dhruvi16] Create sidebar component for the component library. #332
  • [@Dhruvi16] Create tabs component for the component library. #331
  • [@Dhruvi16] Create input component for the component library. #330
  • [@Dhruvi16] Create button component in the component library. #322
  • [@arvind-1] Remove database keys API #313
• New issues created:
  • Table OID is not unique validation error, when creating a table using a data_file #595, created by @pavish
  • New setup fails during the install step #579, created by @pavish
  • Error importing CSV file through the frontend #577, created by @aaditkamat
  • Patch request for schema with same name results in 500 #576, created by @pavish
  • Double quotes in file headers break CSV import #572, created by @eito-fis

05. Editable Tables

• Issues started, not yet finished:
  • [@pavish] Implement record editing #195

06. 2021-09 Stability

• Issues started, not yet finished:
  • [@andyharley-cci] Set up staging environment for Mathesar #225
• New issues created:
  • Set up Vulture to scan for unused Python code #602, created by @kgodey
  • Create helper functions for quoting raw SQL #597, created by @kgodey
  • Refactor db code to work with existing connections #592, created by @kgodey
  • Set up logging in the backend code #583, created by @kgodey
  • Reorganize code in db and set some standards for imports #582, created by @kgodey

07. Working with Tables

• Issues closed:
  • [@eito-fis] DB function to get rows with duplicate values #515
  • [@eito-fis] Add filter to show records with duplicate values in a set of columns #501
  • [@eito-fis] DB and API work for adding a default value to columns #494
  • [@eito-fis] Extend column API to support duplicating columns #452
• Issues started, not yet finished:
  • [@raduab] Frontend work for viewing and deleting table constraints #328
  • [@seancolsen] Implement adding a new empty table #444
  • [@dmos62] Implement showing and changing a column's type #498
  • [@pegnott] Implement adding a new column #495

2021-09 improvements

• Issues closed:
  • [@mr-gabe49] Select component keyboard accessibility #483
  • [@mr-gabe49] Changes to sorting order behavior #474
• Issues started, not yet finished:
  • [@asharonbaltazar] CellHeader dropdown floats above the sidebar #594
  • [@arvind-1] Column API should support creating columns without names  #555
  • [@linzhou-zhong] Uncaught 404s in record viewsets and pagination classes #517
  • [@andyharley-cci] Handle non-unicode encoded datafiles #421
  • [@mr-gabe49] UpArrow does not work in the group section dropdown when data is grouped. #609
• New issues created:
  • UpArrow does not work in the group section dropdown when data is grouped. #609, created by @mr-gabe49
  • Select component: When scrollbar is present within the dropdown, active option should always be visible #608, created by @pavish
  • CellHeader dropdown floats above the sidebar #594, created by @asharonbaltazar
  • Issues with working on a Windows machine #570, created by @asharonbaltazar

08. Initial Data Types

• Issues closed:
  • [@ghislaineguerin] Design for data types that are not "Mathesar types" #537
  • [@eito-fis] Handle DATE data type in the backend #425
  • [@ghislaineguerin] Design for duration data type #241
  • [@ghislaineguerin] Design for date & time data types #240
  • [@ghislaineguerin] Design for money data type #239
  • [@ghislaineguerin] Design for Number data types #238
  • [@ghislaineguerin] Design for boolean data type #237
  • [@ghislaineguerin] Design for URI data type #236
  • [@ghislaineguerin] Design for email data type #235
• Issues started, not yet finished:
  • [@powellc] Handle BOOLEAN data type in the backend #387
  • [@mathemancer] Backend work for Mathesar money custom type  #417
  • [@eito-fis] Handle TIME data type in the backend #426
  • [@powellc] Implement filtering options for Boolean types #388

09. Working with Views

• Issues started, not yet finished:
  • [@ghislaineguerin] Support for opening and interacting with a View #466
  • [@ghislaineguerin] Support for deleting a database view #454
  • [@ghislaineguerin] Design for usage of Foreign Key constraints #243

10. Data Modeling

• Issues closed:
  • [@kgodey] Create design and backend issues for Data Modeling milestone #538

2021-10 improvements

• New issues created:
  • Set up automatic code-coverage reporting for the backend #603, created by @kgodey

11. Multiple Database Support

• Issues started, not yet finished:
  • [@aaditkamat] Backend support for tables without single-column primary keys #395

API v1

• New issues created:
  • PATCH requests to the Table API should support changing the table's name and columns at the same time #593, created by @kgodey