Getting rate limited on PC while bots are running with proxies

[What Comes Around]

Good day,

I have bots that are scraping the CSGO market for new items and their prices. I have around 42 bot accounts doing this, every 5 bots have their own proxy ip. I assign the ip to the steam client. However, my actual ip is getting rate limited even though I don't use it much, so I guess the requests are going through the proxy. Here is the code with which I assign a proxy: 

    let Client = {

        client: new SteamUser({

            httpProxy: proxy,

            webCompatibilityMode: true

        }),

        community: new SteamCommunity(),

        manager: new TradeOfferManager({

            steam: this.client,

            community: this.community,

            language: 'en'

        })

}

Am I not using this correctly? How can I narrow down the issue? I am 100% sure it's because of my bots running, because when I turn them off and wait for a little while, my ip is no longer rate limited.

[Dr. McKay]

You need to set httpProxy on the SteamCommunity instance too.

[What Comes Around]

9 hours ago, Dr. McKay said:

You need to set httpProxy on the SteamCommunity instance too.

Perfect! Thank you very much! I will see if that solves all my issues.

[What Comes Around]

10 hours ago, Dr. McKay said:

You need to set httpProxy on the SteamCommunity instance too.

And is the same thing necessary for TradeOfferManager or does it inherit the proxy from client and community?

[Dr. McKay]

If you pass a SteamCommunity instance to TradeOfferManager, then it inherits it.

[What Comes Around]

21 hours ago, Dr. McKay said:

If you pass a SteamCommunity instance to TradeOfferManager, then it inherits it.

I am still getting rate limited, do you perhaps know any tools to perhaps bind http requests to so I can see which functions are making requests without going through a proxy?

[Dr. McKay]

Wireshark might help

[What Comes Around]

52 minutes ago, Dr. McKay said:

Wireshark might help

Yeah I have tried using wireshark before, but my networking knowledge is very lacking. I think the easiest way to get around this problem for me is to just use VM's that use proxies. Only issue with that is if I want information to converge, I will need to setup a webserver to manage the bots. That's not a big deal because I know how to code websites but I'm just wondering what would take more time, learning and finding the problem or making vm's and a webserver to converge the data. Issue is the longer it takes the more money I lose, so I think I will go with the easier option first, to patch things up, and then look into finding the problem with requests not going through a proxy.

Just a side note; could it be possible that using community.httpRequestPost outside of it's constructor mean it doesn't inherit proxy settings? Because as far as I can tell, all of my bots are logging in from separate ip's (looking at the log in history on steam). And what's being rate limited in specific is getting item data from steam community, which I made my own function for using community.httpRequestPost. By the way, I'd like to mention, I am self taught in coding, because of this I only learned what I wanted/needed, so I'm sorry if I don't explain things very clearly or have naïve ideas.

[Dr. McKay]

Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so:

const Request = require('request');
const SteamCommunity = require('steamcommunity');

let community = new SteamCommunity({
    request: Request.defaults({proxy: 'http://1.2.3.4'})
});

 

[What Comes Around]

47 minutes ago, Dr. McKay said:

Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so:

const Request = require('request');
const SteamCommunity = require('steamcommunity');

let community = new SteamCommunity({
    request: Request.defaults({proxy: 'http://1.2.3.4'})
});

 

Ah ok. Thanks! No worries

I'll try it out!

Edited October 12, 2021 by What Comes Around

[What Comes Around]

3 hours ago, Dr. McKay said:

Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so:

const Request = require('request');
const SteamCommunity = require('steamcommunity');

let community = new SteamCommunity({
    request: Request.defaults({proxy: 'http://1.2.3.4'})
});

 

Good news, works like a charm! Scraping data just fine now!

3 hours ago, Dr. McKay said:

Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so:

const Request = require('request');
const SteamCommunity = require('steamcommunity');

let community = new SteamCommunity({
    request: Request.defaults({proxy: 'http://1.2.3.4'})
});

 

Oh and it makes sense to me now. I looked through steam community at the constructor options and I didn't see any httpProxy options. Now I see there is a request option, but instead I pass in my own request default settings and that gets replaced by my passed in option. Do I understand this correctly?

Edited October 12, 2021 by What Comes Around

[Dr. McKay]

Yep, seems like you understand it.