Library types are too heavy to work with

[orihomie]

I was able to do some kind of task to select about 2.5 thousands workitems (by work item query), and there was one big problem I was tryin to solve - to parse them ASAP.

And that was very ridicilous experience:

1. my idea was to parse them asynchronously and that taken about 2-3 hours for me to get the problem: heavy workitems are hardly to parse asynchronously on multiprocessing pool.

2. And the second idea was to take only fields I've needed then parse them on multiproc pool and that worked perfectly for about 20 seconds on the main processor without any pools.

This is where I've got an idea - to make workitems more 'lighter'.
For now I dont know how to make it real, only thought to make it some kind of Lazy from C#, but could anyone give a suggestion if its real or not and if it's ok to deal with such tasks as I did?

[Boltyk]

@orihomie C#?

What are you doing with all these work items?
What method are you using to get these 2.5 grans of work items?
run_wiql?

[orihomie]

@Boltyk yes, run_wiql, that task was sort of statistics.

I've said kind of C# Lazy type where you can get parts of object not instantly on initialization of object, but on demand of each part you want to get. And this is only idea, I dont know for now if it's good or bad.

[Boltyk]

Not sure I've got the problem.

So you run_wiql and it gives you list of work items.
This list is just a list of work item ids, as I understand.
https://docs.microsoft.com/en-us/azure/devops/integrate/previous-apis/wit/wiql?view=tfs-2015#get-work-items
Supposed to be pretty lightweight

Then you try to get them one by one for some processing.
And there you hit the problem, right?

Was interested by myself about laziness in Python recently.
Found this article
http://derrickgilland.com/posts/late-value-passing-for-lazy-method-chaining-in-python/

[orihomie]

No, it's not just ids, after wiql runs every id turns into WorkItem-type instance.

[Boltyk]

Ough, just having list of these work items as a python workitem objects in a one giant list is a problem itself... Interesting

[orihomie]

It is possible to run that id-workitem casting asynchronously, that would be a bit speed up, but that's not for this issue.

[orihomie]

I was talking about this kind of laziness:
https://pypi.org/project/lazy/

[allburov]

2. And the second idea was to take only fields I've needed then parse them on multiproc pool and that worked perfectly for about 20 seconds on the main processor without any pools.

The issue #14 can help you get a "light" WI.

[Boltyk]

@orihomie did you get a workaround?