Discussion Possible ram usage improvement in module

[Blue]

I was doing some tests in one of my projects that use this module, and i noticed that de polling sometimes requests to much ram, after get further the module in github, the "problem", seems to be that to much information is cached in the LeastUsedCache(), this happens only when there is a that bot account has to many items i suppose..

 

so having that in mind, i have a question that can possibly lower the ram usage, what if we use a mongoDB or something alike to store that data intead(as an option to implement of course), this would only be useful in some cases like mine, but it might improve a little the ram usage, and also would be possible to store more data for more time if needed..

 

i could have did it myself and test it to see if is worth it, but i'd rather see you guys thoughts first, and also the possibility of this to be implemented in the module by the @Doctor himself, if the results seems to be good

[Dr. McKay]

This is already done in the module via persisting to local disk with the dataDirectory option (which you can redirect to go anywhere you want, including MongoDB). Update to v2.10.0 and you can override the maximum size of the asset cache, but beware that doing so will negatively impact performance as more and more items will need to be retrieved from the disk for every poll.

 

In practice, for the best performance, you will need more RAM the more trades you make.

[Blue]

so basicaly if i down a bit the asset cache values, i would use less ram, but the polling would be done slower? this doesnt sound that bad, but how badly this will impact the performace?

[Dr. McKay]

If you're going to lower the max items value, I would also configure a dataDirectory so that excess items can be stored locally on the disk instead of having to consult Steam every time. Performance shouldn't be impacted too heavily, but it may be noticeable. You'll definitely minimize your impact by configuring a dataDirectory.

[Blue]

Im using savePollData:true on the constructor, so my dataDirectory  is already seted right?

[Dr. McKay]

As long as you didn't set dataDirectory: null you should be fine.

[Blue]

ok cool, what about the steamcommunity module, this proccess is the same about the getInventoryContents? i saw that you made a temp var to load the target inventory, but if the target inventory is to big it can pretty much overload the ram as well right? would be cool to have something alike to keep control of ram usage if it doesnt exist yet, also thanks for the quick responses =D

[Dr. McKay]

I have no idea what you're talking about.

[Blue]

// A bit of optimization; objects are hash tables so it's more efficient to look up by key than to iterate an array
	var quickDescriptionLookup = {};

	function getDescription(descriptions, classID, instanceID) {
		var key = classID + '_' + (instanceID || '0'); // instanceID can be undefined, in which case it's 0.

		if (quickDescriptionLookup[key]) {
			return quickDescriptionLookup[key];
		}

		for (var i = 0; i < descriptions.length; i++) {
			quickDescriptionLookup[descriptions[i].classid + '_' + (descriptions[i].instanceid || '0')] = descriptions[i];
		}

		return quickDescriptionLookup[key];
	}

This part of your node-steamcommunity, i meant the "quickDescriptionLookup", if the target inventory has to many descriptions, it could overload ram right?

 

like for instance some target inventory with 100k items or something..

Edited June 21, 2019 by Blue

[Dr. McKay]

Sure, I guess, but there's no way around that. You just need to have enough memory.