Insights wanted on gstore-node! :)

[sebelga]

Hello to all of you using gstore-node!

I'd like to ask you something... What do you think could be done better to make gstore-node more visible/used by Node.js developers that are currently simply using the "google-cloud/datastore" package?

When I see the stats in npm of both packages:

• google-cloud/datastore: 23,304 downloads in the last 7 days
• gstore-node: 400 downloads in the last 7 days

As gstore-node offers exactly the same API + many more features, I do wonder what is the reason for such a difference... Maybe the Datastore is not used/thought with Schemas in mind?

Happy to hear some feedback! 😄

[pauldps]

Datastore schemas are so dynamic, down to the point of defining indexes per record/entity, as opposed to indexes on the entire Table/Kind. So I can create one record with an index in one column, and another record without that index in the same column. So you might be onto something about Datastores not being used with schemas in mind. I see the usefulness in schema validation though.

One thing I'm a little concerned about is memory consumption. I made a simple Google Function using gstore-node and a single request to the API used 80MB of memory. I think it's a bit too much, but I'm not sure whether this is coming from gstore-node or google/datastore, or my own code. For comparison, a Google Function using Sequelize connecting to MySQL uses about 30MB or memory. I didn't have the chance to test things yet as I'm not currently working on APIs and it's going to take a while until I can. But once I have a chance, I'll let you know of my findings.

[sebelga]

@pauloddr Thanks for your insight. I can understand perfectly the need to have this flexibility of dynamic fields. It's the beauty of Schemaless :) But I don't see a use case where you don't want a property being indexed for some entities and for others you do. Don't you expect consistency in the Queries result?

For the memory, if you have time to make the same test with google-cloud/datastore it'd be great! 😄 I don't think, with a few Models (that's what you'd have I guess) gstore adds much memory, but I might be wrong. I will look into this also to make sure. 👍

[trollkotze]

@sebelga A use case for not wanting to index a property for some entities might be properties that can be larger than 1500 bytes on occasion.
Datastore would not be able to index those and throw an error if one has not specified that property to not be indexed for that entity. (But one might want it to be indexed for most other entities anyways.)
But I guess it would still be possible with gstore to set excludeFromIndexes on a per-entity basis? I have not yet probed deeper here. But this lib looks nice! 👍

Does this library offer any kind of functionality to easily keep track of entity counts by any chance? This is a problem I am stuck with at the moment (using sharding counters but running into problems with transactions expiring too quickly under heavy load with lots of updates).

I would really like to try this out. Hope I'll find the time. 🕚 (If this lib would help me with the counting problem, I think that would save me a lot of time. I'm beyond all deadlines atm. 😅 )

[sebelga]

@trollkotze
For the indexes, yes it's possible to excludeFromIndexes for a property but (for now) it "sticks". That means that the next entity with the same schema would have that property excludeFromIndexes set to true.
So, to understand you well, you calculate at runtime the size of a string, and according to the size of the string you decide to exclude it or not from indexes? But then I must ask, why do you index it to begin with?

For your "counts" issue, I have in my todo list (for some time now) the task to add an abstraction to automatically shard entities with a counter built-in :) But I haven't had the chance to work on it yet. Can I ask what do you mean by "transactions expiring too quickly"?
For now, one solution I see is using a Redis db for the counter. With gstore-node it's very easy with it's cache (https://sebelga.gitbooks.io/gstore-node/content/cache.html).

const gstore = require('gstore-node')();
const { cache } = gstore;

// read
cache.get('user-count').then(...);

// write
cache.set('user-count', 12).then(...);

[sebelga]

Hi @trollkotze
Sorry for coming back late on this. Do you mind creating a new issue with this as a question so this thread does not change the focus?

You are probably doing something wrong as 60 seconds is a lot 😄 But it'd be better to get help and insight in a separate issue ticket.

Just for info https://redislabs.com/ has a free tier of 30MB which is plenty for a counter 👍

Cheers!

[trollkotze]

Hi @trollkotze
Sorry for coming back late on this. Do you mind creating a new issue with this as a question so this thread does not change the focus?

Yes, I had that on mind, now especially after having looked deeper into your library, but deciding not to use it, because I see a lot to improve.

I have not much time at the moment to get deeper into this and provide constructive criticism or ideas, because I have to get finished with something. But it has helped me to find into Redis at least, which I am now using besides Cloud Datastore, and I hope to come back here when I find time, because I really like the idea of this project.

And yes, I am already using RedisLabs free tier now for testing, will use enterprise tier in production as well. Thanks for the tip!

(It seems I cannot delete my long rambling comment from here, after I just moved the issue over to here. I guess it makes sense to keep this thread clean, so feel free to remove it here.)

[sebelga]

@trollkotze

"but deciding not to use it, because I see a lot to improve."

Looking forward for your insight and how we could improve the lib. Whenever you have time! 👍

[carnun]

@sebelga I'm trying to remember how I first found your package... I think it was via mongoose and searching for something similar for Google Datastore. It was just a lucky find on my part, otherwise I would have gone the raw @google-cloud/datastore route. I suspect the download stats are more a visibility thing, potentially having other people blogging about using it and their use cases may change that.

I use Datastore as a backend for GraphQL APIs and gstore-node fits really nicely with my stack, especially since I heavily use TypeScript. One of the items on my todo list is to create a sequence of blogs that discuss my experiences with this stack (Gstore-node, Apollo, TypeGraphQL, GCF, Datastore) so once I get round to doing that I'll ping you.

[sebelga]

Thanks for the input @carnun ! Yes, I do think the visibility of gstore might be improved, I wish I had more time to blog about it. Let me know about your blog posts when you have them, I am interested to know more about your stack (didn't know about TypeGraphQL, very interesting).