What Comes Around Posted October 7, 2021 Report Share Posted October 7, 2021 Good day, I have bots that are scraping the CSGO market for new items and their prices. I have around 42 bot accounts doing this, every 5 bots have their own proxy ip. I assign the ip to the steam client. However, my actual ip is getting rate limited even though I don't use it much, so I guess the requests are going through the proxy. Here is the code with which I assign a proxy: let Client = { client: new SteamUser({ httpProxy: proxy, webCompatibilityMode: true }), community: new SteamCommunity(), manager: new TradeOfferManager({ steam: this.client, community: this.community, language: 'en' }) } Am I not using this correctly? How can I narrow down the issue? I am 100% sure it's because of my bots running, because when I turn them off and wait for a little while, my ip is no longer rate limited. Quote Link to comment Share on other sites More sharing options...
Dr. McKay Posted October 9, 2021 Report Share Posted October 9, 2021 You need to set httpProxy on the SteamCommunity instance too. What Comes Around 1 Quote Link to comment Share on other sites More sharing options...
What Comes Around Posted October 9, 2021 Author Report Share Posted October 9, 2021 9 hours ago, Dr. McKay said: You need to set httpProxy on the SteamCommunity instance too. Perfect! Thank you very much! I will see if that solves all my issues. Quote Link to comment Share on other sites More sharing options...
What Comes Around Posted October 9, 2021 Author Report Share Posted October 9, 2021 10 hours ago, Dr. McKay said: You need to set httpProxy on the SteamCommunity instance too. And is the same thing necessary for TradeOfferManager or does it inherit the proxy from client and community? Quote Link to comment Share on other sites More sharing options...
Dr. McKay Posted October 10, 2021 Report Share Posted October 10, 2021 If you pass a SteamCommunity instance to TradeOfferManager, then it inherits it. Quote Link to comment Share on other sites More sharing options...
What Comes Around Posted October 11, 2021 Author Report Share Posted October 11, 2021 21 hours ago, Dr. McKay said: If you pass a SteamCommunity instance to TradeOfferManager, then it inherits it. I am still getting rate limited, do you perhaps know any tools to perhaps bind http requests to so I can see which functions are making requests without going through a proxy? Quote Link to comment Share on other sites More sharing options...
Dr. McKay Posted October 12, 2021 Report Share Posted October 12, 2021 Wireshark might help What Comes Around 1 Quote Link to comment Share on other sites More sharing options...
What Comes Around Posted October 12, 2021 Author Report Share Posted October 12, 2021 52 minutes ago, Dr. McKay said: Wireshark might help Yeah I have tried using wireshark before, but my networking knowledge is very lacking. I think the easiest way to get around this problem for me is to just use VM's that use proxies. Only issue with that is if I want information to converge, I will need to setup a webserver to manage the bots. That's not a big deal because I know how to code websites but I'm just wondering what would take more time, learning and finding the problem or making vm's and a webserver to converge the data. Issue is the longer it takes the more money I lose, so I think I will go with the easier option first, to patch things up, and then look into finding the problem with requests not going through a proxy. Just a side note; could it be possible that using community.httpRequestPost outside of it's constructor mean it doesn't inherit proxy settings? Because as far as I can tell, all of my bots are logging in from separate ip's (looking at the log in history on steam). And what's being rate limited in specific is getting item data from steam community, which I made my own function for using community.httpRequestPost. By the way, I'd like to mention, I am self taught in coding, because of this I only learned what I wanted/needed, so I'm sorry if I don't explain things very clearly or have naïve ideas. Quote Link to comment Share on other sites More sharing options...
Dr. McKay Posted October 12, 2021 Report Share Posted October 12, 2021 Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so: const Request = require('request'); const SteamCommunity = require('steamcommunity'); let community = new SteamCommunity({ request: Request.defaults({proxy: 'http://1.2.3.4'}) }); Quote Link to comment Share on other sites More sharing options...
What Comes Around Posted October 12, 2021 Author Report Share Posted October 12, 2021 (edited) 47 minutes ago, Dr. McKay said: Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so: const Request = require('request'); const SteamCommunity = require('steamcommunity'); let community = new SteamCommunity({ request: Request.defaults({proxy: 'http://1.2.3.4'}) }); Ah ok. Thanks! No worries I'll try it out! Edited October 12, 2021 by What Comes Around Quote Link to comment Share on other sites More sharing options...
What Comes Around Posted October 12, 2021 Author Report Share Posted October 12, 2021 (edited) 3 hours ago, Dr. McKay said: Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so: const Request = require('request'); const SteamCommunity = require('steamcommunity'); let community = new SteamCommunity({ request: Request.defaults({proxy: 'http://1.2.3.4'}) }); Good news, works like a charm! Scraping data just fine now! 3 hours ago, Dr. McKay said: Sorry, I misled you. You need to pass a request instance with the proxy set on it to SteamCommunity, like so: const Request = require('request'); const SteamCommunity = require('steamcommunity'); let community = new SteamCommunity({ request: Request.defaults({proxy: 'http://1.2.3.4'}) }); Oh and it makes sense to me now. I looked through steam community at the constructor options and I didn't see any httpProxy options. Now I see there is a request option, but instead I pass in my own request default settings and that gets replaced by my passed in option. Do I understand this correctly? Edited October 12, 2021 by What Comes Around Quote Link to comment Share on other sites More sharing options...
Dr. McKay Posted October 12, 2021 Report Share Posted October 12, 2021 Yep, seems like you understand it. What Comes Around 1 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.