Jump to content
McKay Development

Getting rate limited on PC while bots are running with proxies


Recommended Posts

Good day,

I have bots that are scraping the CSGO market for new items and their prices. I have around 42 bot accounts doing this, every 5 bots have their own proxy ip. I assign the ip to the steam client. However, my actual ip is getting rate limited even though I don't use it much, so I guess the requests are going through the proxy. Here is the code with which I assign a proxy: 

    let Client = {
        client: new SteamUser({
            httpProxy: proxy,
            webCompatibilityMode: true
        }),
        community: new SteamCommunity(),
        manager: new TradeOfferManager({
            steam: this.client,
            community: this.community,
            language: 'en'
        })
}

Am I not using this correctly? How can I narrow down the issue? I am 100% sure it's because of my bots running, because when I turn them off and wait for a little while, my ip is no longer rate limited.

Link to comment
Share on other sites

52 minutes ago, Dr. McKay said:

Wireshark might help

Yeah I have tried using wireshark before, but my networking knowledge is very lacking. I think the easiest way to get around this problem for me is to just use VM's that use proxies. Only issue with that is if I want information to converge, I will need to setup a webserver to manage the bots. That's not a big deal because I know how to code websites but I'm just wondering what would take more time, learning and finding the problem or making vm's and a webserver to converge the data. Issue is the longer it takes the more money I lose, so I think I will go with the easier option first, to patch things up, and then look into finding the problem with requests not going through a proxy.

Just a side note; could it be possible that using community.httpRequestPost outside of it's constructor mean it doesn't inherit proxy settings? Because as far as I can tell, all of my bots are logging in from separate ip's (looking at the log in history on steam). And what's being rate limited in specific is getting item data from steam community, which I made my own function for using community.httpRequestPost. By the way, I'd like to mention, I am self taught in coding, because of this I only learned what I wanted/needed, so I'm sorry if I don't explain things very clearly or have naïve ideas.

Link to comment
Share on other sites

47 minutes ago, Dr. McKay said:

Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so:

const Request = require('request');
const SteamCommunity = require('steamcommunity');

let community = new SteamCommunity({
    request: Request.defaults({proxy: 'http://1.2.3.4'})
});

 

Ah ok. Thanks! No worries :)

I'll try it out!

Edited by What Comes Around
Link to comment
Share on other sites

3 hours ago, Dr. McKay said:

Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so:

const Request = require('request');
const SteamCommunity = require('steamcommunity');

let community = new SteamCommunity({
    request: Request.defaults({proxy: 'http://1.2.3.4'})
});

 

Good news, works like a charm! Scraping data just fine now!

3 hours ago, Dr. McKay said:

Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so:

const Request = require('request');
const SteamCommunity = require('steamcommunity');

let community = new SteamCommunity({
    request: Request.defaults({proxy: 'http://1.2.3.4'})
});

 

Oh and it makes sense to me now. I looked through steam community at the constructor options and I didn't see any httpProxy options. Now I see there is a request option, but instead I pass in my own request default settings and that gets replaced by my passed in option. Do I understand this correctly?

Edited by What Comes Around
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
×
×
  • Create New...