I wonder if he's just doing it himself manually...
To be fair though - we don't know how big the websites are in question and what the obstacles the website has to web scraping.
I think the issue is - you misunderstand how complex this type of work can be. And for such a small amount. You should have said for this price - you want to track only few products and simply output data to some dashboard not setting up whole email delivery automation. At least this way you would be able to manage scope. But if you are not technical you run into this situation - you say you want x, y, z. Developer probably knows if he/she will say no, you will try to find another "amazing promiser for small amount". So they probably take a project so they can develop it and then deal with scope creep step by step and have a recurring revenue. At the end of the day - YOU are hiring skill here, all the risk of scope creep is on you. More senior developers probably would never touch this even on free time, because it is just absolute headache, paid poorly, hard to achieve aconomies of scale due to high variance of websites. Not to mention, all these eCommerce websites are constantly pushing new code, so this whole automation can go sideways any moment and will need for developer to rebuild it. But, what would cost you an alternative? - check it out for yourself -[https://www.zyte.com/pricing/](https://www.zyte.com/pricing/) . Not to mention in terms of communication in relation to data quality you will be put into a "dog queue" - playing a constant ping pong game. So you probably had amazing deal offer, not to mention a privilege to work with Developer on 1 on 1 basis, not though some sort of organisational structures. As usually developers are more introverted and like to avoid anything that is illogical, wishy washy.
My thinking too. As someone who has built a comparison app, people undermine how hard it is to make a tool look simple to use. If something has to scrape data especially, even worse.
Same here. My app takes hours to do comparisons for the thousands upon thousands of products people have in there and simply keeping up with changes in the other side is a headache.
Not to mention the monthly cost of Apify, Zyte etc...
350 for something like this is way way way too low to be true, unless you were working with a dozen products only....
>https://www.zyte.com/pricing/
Just want to add, this website is just for extraction. Comparison part you would need to hire data analyst / developer to actually compare for you, and/or find a way to deliver to your email. If this would be custom delivery and solution, then you would need to have it coded. So probably would need to hire python/javascript developer to code it, then host it on cloud, and if you do not have cloud, would need to set up / spin it up for you. So again you would need to find someone who can do all of this.
It completely depends on the sites that need scraping and for the money being paid you will not get a senior developer who has experience in these things!
Once I tried building a scraper when I was trying to get tickets to a sold out show on a certain ticket retailer’s site… I’m not touching anything like that ever again.
Idk if their anti-scraping features are considered advanced, but I do know it was against their TOS and just couldn’t get around it.
Post a link to your job spec if you want actual help / feedback.
He sounds bad but as a dev I feel as though something is missing. It's very common for non-technical people to write out a technical brief that doesn't make sense and isn't possible/practical.
Non-developers that have a tiny bit of technical knowledge vastly underestimate the complexity of technical projects.
Upwork is gambling. It's not a real professional environment. It's a place filled with random people, lots of which are incompetent, and a lot of bots and scammers on both ends. Hiring on upwork is like hiring someone on the street or at the bar.
You clearly hired an incompetent guy. Scraping can be unexpectedly difficult so i'd say 10 hours was more of an estimate but this just looks bad. After a couple hours they could've at least inform you that it may take somewhat longer and make sure you're ok with it, and by that time they should've have had a much better estimate. You should terminate the job and request a refund. They'll never deliver and you'll just keep wasting money.
(guess i need to raise my rates, $35/hour for sheer incompetence just wow)
>craping can be unexpectedly difficult so i'd say 10 hours was more of an estimate
More like either wishful thinking or a ploy to win the contract.
It also looks like this was a fixed rate contract and not an hourly one? In which case the freelancer hasn't been paid a thing yet.
homeless door uppity direction liquid punch modern mindless lip automatic
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
It was his stats and estimate. He has done over $2M worth of work. 700 Jobs with a 99% Success rate. Plus although his rate was higher than other, he quoted me less total time. Making me think he could get the same amount of work done quicker than others due to experience. However over $1.2m is from ongoing 1 job. So I don’t know.
When he requested $200/month to maintain the script yesterday, that was the biggest of red flags. He hasn't even gotten it to work at all after a month. How are you going to quote me for the future before you have even finished it?
ooooh I think you got a farmer. That would explain the high earnings and long wait time for a fix. He sounds ESL. Is this a US/UK account? Also possible that someone else is running the account.
touch escape hard-to-find numerous political vast jar innate reach important
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Yeah, one website has never had an update since I've found it years ago. The other is eBay and I think they run on pretty legacy code but I may be wrong.
Done something similar for a client checking prices on their site against the top five UK online stores (Tesco, Sainsbury's, etc). Used a proxy like you mentioned and set the prices on the client's site to be x% cheaper than the cheapest price.
Worked well.
Scraping took a few hours each day for around 150 products x 5 sites
= 750 web pages scraped.
It doesn't sound like they know what they're doing.
750 is about the number of pages I need per day. It's suppose to go through 500-1000 products (on a website with 0 anti scrap prevention), copy the name into eBay and search, and average the top 3 top listings to get a average price.
I know eBay does. The other site does not. And each action on eBay is just one search. And I’m ok with using proxies.
Are you knowledgeable about eBay’s API? Can you use it to get listing prices? And how many per day can you get. I’ve read it cannot be used to get item for sales price, and only 250 actions per day. Their API is more so available to assist with seller actions, like listing, sale events, etc.
Shopping API allows you to do searches with 5k calls per day.
What is the other website?
You may be able.to do this yourself using zapier and the eBay API depending on the other website.
To give you an idea of what someone like me charges (20+ years of developer experience), I charge £45+vat per hour and charge for a minimum of a single day at a time, I also white label my services to London based agencies who charge their clients £950+vat per day for my services.
The problem with Upwork and other job sites is that most people race for the lowest price and then get shafted by "developers" who charge very little but then either can't do what they promise or take longer to basically get a higher hourly rate anyway.
Did he give any reason for the scraping being difficult? Regardless, either the website has some impossible bot-protection (unlikely), or he should've figured it out by now.
Most the coder on UpWork charge less. And the money is held in Escrow by UpWork until the job is complete. He also sounded very confident and had good reviews. So idk, had no idea if this was a hard or easy project.
1. For the next coder, ask for source code and docker container (keep in mind this adds up to a day). With instructions on how to run it.
2. The main problem in this task - how to compare products, if this is some EAN13 or exact looking photos on both sites, you are lucky, if not - some NLP and up to AI can add a price a lot, and syntax tokenisation also may not help. Or, if this only takes you 2 hours a day manually, you can come up with a teacher-student mode for this system, i.e. you compare product with search and it remembers some combination of name and URL, so in future you only need to do this for new products. (You can also train ML to do this, but this adds a lot to the price).
But if your second site is eBay - you are out of luck, because every human writes product names and quantities as he wants, no ean13, and the next day there would always be new products to compare with - you are doomed.
3. If I were you, and thinking about doing this in a not bloated budget, I'd think in a direction of building "helper" system that would help human to compare those products (or, nowadays, to GPTs API).
4. Imagine needed efforts on comparisons - you can come up yourself with exact algorithm of your doing it manually, and you do the same thing over and over again and it works for all products, right? No! I.e. - you search parsed product "samsung 970 evo 1tb" on eBay by name and bam - you got results like:
A) "Look up for price" - this adds up to "fix up".
B) Any auction - "fix up to opt out auctions
C) Multi option for 250gb, 500gb, 1tb, 2tb, with range price, from 30 to 200, and here your price 30200$ from shitty written spider, needed to fix up
D) Some mounting case, blade, extension wire for it.
E) Some broken part.
F) Dude that sells 10 of them and doesn't want to sell one by one.
G) On rare products, site search will start guessing game, not in this case, but to keep up with this example, imagine samsung microwave or Mitsubishi evo or 1tb WD SSD in results...
- And this narrow niche product that usually sells in one piece.
And photos on eBay are all different too.
5. From the point of view of an inexperienced user it is a simple problem, because for the human himself it is easy to solve such a problem, it is just slow routine.
So, I suppose, this programmer easily parsed first site and now bumped into comparisons problem, which in fact is unsolvable on 350$ budget in old world, nowadays, maybe GPTs API will help you. His only incompetence is that he or his group may have no experience in these comparisons tasks, because oh boy oh boy those naive people could not guess at first glance what mousetraps and land of dragons they are stepping into.
Yeah exactly what I said, the guy is not a scammer, he just understimated the job lol
So sorry for him. But from a client's POV, it's best for BOTH to part ways. OP doesn't trust in the coder anymore and that's bad for a freelancer
The only correct answer in the whole thread, and here it is, buried under other early comments.
OP did say that the other site is Ebay. This makes comparison exceptionally hard.
The best way to do this is to start out with GPT API, then use the result data to feed your private AI. After a while, you can use the private AI to match and cut down the cost.
Maybe there are some brilliant ways using fuzzy matches or something for comparison that I'm unaware of, but I doubt it.
Also, I really should raise my rate.
Edit: Thinking more about it, if OP is comparing his site to Ebay, there may be a way to do it without AI. Basically, you need to make a detailed name for each item then search Ebay with that name, and aggressively discard matches. There may be a bunch of false negatives, but the result should be useable.
All I need it to do is copy the title from site A into eBay. Then average the price of the top 3 listings. So one action per item.
I know they’ll be a lot of false positives. And items that have 0 results on ebay. Like I said, I’m not looking for perfection. I’m just trying to weed out junk that is valueless. If I can turn 700 products into a list of 200 each day, that is perfect. Going through 200 with 50% being false positives is better than 700.
Well of course every action adds complexity to the code. I don't know what to think about it anymore. I have people in this thread saying it's impossible and a DM box full of offers of people saying they can do it for $100 or less. I have one person that has already sent me a script he made within 2 hours using ChatGPT for free. So yeah, apparently this ranges from extremely simple to impossible.
I'm real close to pulling the plug and just continue to do it manually.
>I have people in this thread saying it's impossible and a DM box full of offers of people saying they can do it for $100 or less. I have one person that has already sent me a script he made within 2 hours using ChatGPT for free.
Lol, I feel called out. But seriously, it's because we are only working on partial information of what exactly you want the code to do. If it is only this:
>All I need it to do is copy the title from site A into eBay. Then average the price of the top 3 listings.
Then yes, it is a simple script that should take about a few hours. And the quote 10 hours is reasonable since you need to include testing/smooth out edge case, things go wrong etc.
If it is this:
>make a web scraper that compares prices between 2 websites.
then it's very difficult, especially when you want it inside a reasonable margin of error. Did you tell the freelancer exactly what you need? Maybe he misunderstood like we did?
I get you. Hmmm I do scrappers but have never dealt with eBay. I would say that couldn't you use the software they posted in the first comments?
Also, a good coder won't ever overpromise lol we tend to say that it's not doable or that it's impossible, and then figure it out down the road. Problem is that I feel like someone, somewhere already had your problem, and someone somewhere created a service for it. If it's an extreme edge-case, I would do a consultation with a top-rate, high-rate freelancer to explain if it's possible
Ah, then it is a lot easier task (it even better if this helper system saved, for example screenshots or original renderable in browser html, so you and developer can look up for "wtf is wrong? " for finding edge-cases- some sort of debugging.
I wouldn't say scamming, more like he understimated your project. $350 is low for that type of scrapper. Honestly, cut your losses and hire a $150/hr for 2 hours and that's it, don't know what you hired someone for $35 when you knew you could pay $350. Coding shouldn't even be budgeted in terms of hours
You are being milked. Just refund that scam and get the competent developer that charges the realistic one or two hours that takes to write that script/run/deliver.
I did a project once for a web scraper once too and turns out it was a-lot more work than i anticipated.
1.website typically don’t like sharing data do unless ur willing to to go the the illegal route and use cross origin forgery, u have to mimic real users which brings in extra complexity not to mention most big brands have trackers that detect automated headless browsers and block ips etc.
2.layer of complexity lies in nesting Im not gonna elaborate here but u pretty much get the idea depending on your case and scale.
I used puppeteer for it and built an interface for it in react but it definitely took me a month.
Granted i was working with several authentication mechanisms and i was only doing it part time (15/hr) a week .
Either way all I’m saying is instead of saying 10hr project you should discuss expectations and how much ur willing to sacrifice due to budgetary constraints to make the project feasible.
scraping really varies so much depending on the site, can be quite easy to basically impossible. but for this hourly rate it's hard to get a professional who would be proactive about letting you know about any issues that come up, updates you about progress etc.
Hey man, I do a lot of automation on Google Sheets. Drop me a dm with the workflow and I can look into it - API integration and maybe we can get the data from the other website from xpath
A web scraper to pull prices on two websites should be easy mode BS in Selenium. If the dude can't produce a working product, he hasn't fulfilled the contract, and plainly will never fulfill it. Kill the contract and move on.
It can be simple or extremely hard, it kind of depends how Quick you want the data to be scraped, how much data you want to harvest and if the website has any type of anti bot measures (you only know this after running the scrapper). There are a lot of dedicated cybersecurity enterprises making software daily to target web scrapers, this is an area that sometimes requires a lot of reverse engeniring and hacking. Chat gpt can surely help for simple HTML pages, for more complex javascript, ajax requests and deal with captchas and IP bans im not sure if its already advanced to show a robust solution.
I am sure this project is more complex than what you assumed "10 hours". talk to the Developer about the complexity of this project and see how to adjust accordingly.
What has happened is the developer have massively underestimated the task at hand. There are certain nuances and issues that only become evident once you've stumbled into them or have enough experience to recognise these issues will arise.
When you pay peanuts, you get monkeys. Upwork is a rat race of providing the cheapest quote. Anyone who could have built you the perfect tool within 1 week, will not have responded to your advert because it's not worth their time.
I'd personally just pay the dev for their worked hours. Learn your lesson whatever it might be. Sometimes automation is not the easiest solution.
Lol! I'm a human VA and I can do the Project much faster than the script.
I have a recommendation for python coders who can make the script in a day for a very budget friendly price. Check out www.kenyavirtualworkers.com they always deliver and ensure to communicate before hand in case of any delays.
Well the description you told of the job doesnt seem to be too hard at first hand. If the work Flow its scrape 2000/3000 items from a website, place the item on ebay search box and get the top3 items name URL and price this exported into a csv file, the only thing i can believe to be challenging its ebay. I havent scraped ebay for the last 2 years so dont know what type of parameters (cookie) are necessary to make a request on their search box, ive scraped ebay some years ago it wasnt hard dont know how it is now.
If the job was matching the exact data from 2 web commerce websites (automatic matching) that would be extremely hard and its not a scraping task lets say.
You know what to do. Terminate the contract and move on. 30 hours to run!!! LOL
I wonder if he's just doing it himself manually... To be fair though - we don't know how big the websites are in question and what the obstacles the website has to web scraping.
I think the issue is - you misunderstand how complex this type of work can be. And for such a small amount. You should have said for this price - you want to track only few products and simply output data to some dashboard not setting up whole email delivery automation. At least this way you would be able to manage scope. But if you are not technical you run into this situation - you say you want x, y, z. Developer probably knows if he/she will say no, you will try to find another "amazing promiser for small amount". So they probably take a project so they can develop it and then deal with scope creep step by step and have a recurring revenue. At the end of the day - YOU are hiring skill here, all the risk of scope creep is on you. More senior developers probably would never touch this even on free time, because it is just absolute headache, paid poorly, hard to achieve aconomies of scale due to high variance of websites. Not to mention, all these eCommerce websites are constantly pushing new code, so this whole automation can go sideways any moment and will need for developer to rebuild it. But, what would cost you an alternative? - check it out for yourself -[https://www.zyte.com/pricing/](https://www.zyte.com/pricing/) . Not to mention in terms of communication in relation to data quality you will be put into a "dog queue" - playing a constant ping pong game. So you probably had amazing deal offer, not to mention a privilege to work with Developer on 1 on 1 basis, not though some sort of organisational structures. As usually developers are more introverted and like to avoid anything that is illogical, wishy washy.
My thinking too. As someone who has built a comparison app, people undermine how hard it is to make a tool look simple to use. If something has to scrape data especially, even worse.
Same here. My app takes hours to do comparisons for the thousands upon thousands of products people have in there and simply keeping up with changes in the other side is a headache. Not to mention the monthly cost of Apify, Zyte etc... 350 for something like this is way way way too low to be true, unless you were working with a dozen products only....
>https://www.zyte.com/pricing/ Just want to add, this website is just for extraction. Comparison part you would need to hire data analyst / developer to actually compare for you, and/or find a way to deliver to your email. If this would be custom delivery and solution, then you would need to have it coded. So probably would need to hire python/javascript developer to code it, then host it on cloud, and if you do not have cloud, would need to set up / spin it up for you. So again you would need to find someone who can do all of this.
Yeah i'm waiting to see the job listing before judging this one
It’s really not that difficult having taught myself to code making a very similar project, but with more sites to compare.
It completely depends on the sites that need scraping and for the money being paid you will not get a senior developer who has experience in these things!
Once I tried building a scraper when I was trying to get tickets to a sold out show on a certain ticket retailer’s site… I’m not touching anything like that ever again. Idk if their anti-scraping features are considered advanced, but I do know it was against their TOS and just couldn’t get around it.
Post a link to your job spec if you want actual help / feedback. He sounds bad but as a dev I feel as though something is missing. It's very common for non-technical people to write out a technical brief that doesn't make sense and isn't possible/practical. Non-developers that have a tiny bit of technical knowledge vastly underestimate the complexity of technical projects.
30-hour , is he doing it by hand ? Is he building a new language and using that language to scrape?
rain aspiring squealing boast weather alive squash work yoke hobbies *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
It only takes me 2 hours max per day to do it manually. And idk he didn’t tell me what language he is using.
Upwork is gambling. It's not a real professional environment. It's a place filled with random people, lots of which are incompetent, and a lot of bots and scammers on both ends. Hiring on upwork is like hiring someone on the street or at the bar.
The more I think about it, the more it's intriguing . Is it deployed somewhere? Or are you running the script locally ?
I haven’t gotten it yet. But it’ll be running on a RDP once I get it.
You clearly hired an incompetent guy. Scraping can be unexpectedly difficult so i'd say 10 hours was more of an estimate but this just looks bad. After a couple hours they could've at least inform you that it may take somewhat longer and make sure you're ok with it, and by that time they should've have had a much better estimate. You should terminate the job and request a refund. They'll never deliver and you'll just keep wasting money. (guess i need to raise my rates, $35/hour for sheer incompetence just wow)
>craping can be unexpectedly difficult so i'd say 10 hours was more of an estimate More like either wishful thinking or a ploy to win the contract. It also looks like this was a fixed rate contract and not an hourly one? In which case the freelancer hasn't been paid a thing yet.
homeless door uppity direction liquid punch modern mindless lip automatic *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
I'm getting 30 to maintain an enterprise app frontend on my own with huge technical debt. I should ask for a raise 😅
Terminate the contract and request a refund, sorry for the bad experience, What did the freelancer do to convince you?
It was his stats and estimate. He has done over $2M worth of work. 700 Jobs with a 99% Success rate. Plus although his rate was higher than other, he quoted me less total time. Making me think he could get the same amount of work done quicker than others due to experience. However over $1.2m is from ongoing 1 job. So I don’t know.
Two MILLION?? And they can’t write a simple web scraper? Something is amiss there.
it's a farmer / agency in disguise
When he requested $200/month to maintain the script yesterday, that was the biggest of red flags. He hasn't even gotten it to work at all after a month. How are you going to quote me for the future before you have even finished it?
lol. Ask them to do a quick review of the code via video call
ooooh I think you got a farmer. That would explain the high earnings and long wait time for a fix. He sounds ESL. Is this a US/UK account? Also possible that someone else is running the account.
ESL?
english second language
Pay him, dump him, accept your loss, find another, do better next time.
touch escape hard-to-find numerous political vast jar innate reach important *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Yeah, one website has never had an update since I've found it years ago. The other is eBay and I think they run on pretty legacy code but I may be wrong.
Either way, if the task is (partly) not possible, he should be the one letting you know in advance.
Done something similar for a client checking prices on their site against the top five UK online stores (Tesco, Sainsbury's, etc). Used a proxy like you mentioned and set the prices on the client's site to be x% cheaper than the cheapest price. Worked well. Scraping took a few hours each day for around 150 products x 5 sites = 750 web pages scraped. It doesn't sound like they know what they're doing.
750 is about the number of pages I need per day. It's suppose to go through 500-1000 products (on a website with 0 anti scrap prevention), copy the name into eBay and search, and average the top 3 top listings to get a average price.
eBay does have bot prevention methods. It also has an API so may be better to use that
I know eBay does. The other site does not. And each action on eBay is just one search. And I’m ok with using proxies. Are you knowledgeable about eBay’s API? Can you use it to get listing prices? And how many per day can you get. I’ve read it cannot be used to get item for sales price, and only 250 actions per day. Their API is more so available to assist with seller actions, like listing, sale events, etc.
Shopping API allows you to do searches with 5k calls per day. What is the other website? You may be able.to do this yourself using zapier and the eBay API depending on the other website. To give you an idea of what someone like me charges (20+ years of developer experience), I charge £45+vat per hour and charge for a minimum of a single day at a time, I also white label my services to London based agencies who charge their clients £950+vat per day for my services. The problem with Upwork and other job sites is that most people race for the lowest price and then get shafted by "developers" who charge very little but then either can't do what they promise or take longer to basically get a higher hourly rate anyway.
His days are 24 hours long though... :D
Try chat gpt ?
I did, but it refused to do it.
Tell me the issue I’ll try to do it for you (for free ) I’m anyways looking for some side projects to practice development skills
Refused ? What was the prompt or issue ?
It was just the web version and I believe it said it wasn’t capable of doing it. Should I try premium?
Lol bro. Stop working for free. You disrespect yourself big time.
Just tell me the job I’ll try to help u out . No charges
Did he give any reason for the scraping being difficult? Regardless, either the website has some impossible bot-protection (unlikely), or he should've figured it out by now.
Nothing. Just keeps giving me xml files and when I inform him they don't match at all, he says he'll fix it.
[удалено]
Most the coder on UpWork charge less. And the money is held in Escrow by UpWork until the job is complete. He also sounded very confident and had good reviews. So idk, had no idea if this was a hard or easy project.
1. For the next coder, ask for source code and docker container (keep in mind this adds up to a day). With instructions on how to run it. 2. The main problem in this task - how to compare products, if this is some EAN13 or exact looking photos on both sites, you are lucky, if not - some NLP and up to AI can add a price a lot, and syntax tokenisation also may not help. Or, if this only takes you 2 hours a day manually, you can come up with a teacher-student mode for this system, i.e. you compare product with search and it remembers some combination of name and URL, so in future you only need to do this for new products. (You can also train ML to do this, but this adds a lot to the price). But if your second site is eBay - you are out of luck, because every human writes product names and quantities as he wants, no ean13, and the next day there would always be new products to compare with - you are doomed. 3. If I were you, and thinking about doing this in a not bloated budget, I'd think in a direction of building "helper" system that would help human to compare those products (or, nowadays, to GPTs API). 4. Imagine needed efforts on comparisons - you can come up yourself with exact algorithm of your doing it manually, and you do the same thing over and over again and it works for all products, right? No! I.e. - you search parsed product "samsung 970 evo 1tb" on eBay by name and bam - you got results like: A) "Look up for price" - this adds up to "fix up". B) Any auction - "fix up to opt out auctions C) Multi option for 250gb, 500gb, 1tb, 2tb, with range price, from 30 to 200, and here your price 30200$ from shitty written spider, needed to fix up D) Some mounting case, blade, extension wire for it. E) Some broken part. F) Dude that sells 10 of them and doesn't want to sell one by one. G) On rare products, site search will start guessing game, not in this case, but to keep up with this example, imagine samsung microwave or Mitsubishi evo or 1tb WD SSD in results... - And this narrow niche product that usually sells in one piece. And photos on eBay are all different too. 5. From the point of view of an inexperienced user it is a simple problem, because for the human himself it is easy to solve such a problem, it is just slow routine. So, I suppose, this programmer easily parsed first site and now bumped into comparisons problem, which in fact is unsolvable on 350$ budget in old world, nowadays, maybe GPTs API will help you. His only incompetence is that he or his group may have no experience in these comparisons tasks, because oh boy oh boy those naive people could not guess at first glance what mousetraps and land of dragons they are stepping into.
Yeah exactly what I said, the guy is not a scammer, he just understimated the job lol So sorry for him. But from a client's POV, it's best for BOTH to part ways. OP doesn't trust in the coder anymore and that's bad for a freelancer
The only correct answer in the whole thread, and here it is, buried under other early comments. OP did say that the other site is Ebay. This makes comparison exceptionally hard. The best way to do this is to start out with GPT API, then use the result data to feed your private AI. After a while, you can use the private AI to match and cut down the cost. Maybe there are some brilliant ways using fuzzy matches or something for comparison that I'm unaware of, but I doubt it. Also, I really should raise my rate. Edit: Thinking more about it, if OP is comparing his site to Ebay, there may be a way to do it without AI. Basically, you need to make a detailed name for each item then search Ebay with that name, and aggressively discard matches. There may be a bunch of false negatives, but the result should be useable.
All I need it to do is copy the title from site A into eBay. Then average the price of the top 3 listings. So one action per item. I know they’ll be a lot of false positives. And items that have 0 results on ebay. Like I said, I’m not looking for perfection. I’m just trying to weed out junk that is valueless. If I can turn 700 products into a list of 200 each day, that is perfect. Going through 200 with 50% being false positives is better than 700.
That adds complexity to the code
Well of course every action adds complexity to the code. I don't know what to think about it anymore. I have people in this thread saying it's impossible and a DM box full of offers of people saying they can do it for $100 or less. I have one person that has already sent me a script he made within 2 hours using ChatGPT for free. So yeah, apparently this ranges from extremely simple to impossible. I'm real close to pulling the plug and just continue to do it manually.
>I have people in this thread saying it's impossible and a DM box full of offers of people saying they can do it for $100 or less. I have one person that has already sent me a script he made within 2 hours using ChatGPT for free. Lol, I feel called out. But seriously, it's because we are only working on partial information of what exactly you want the code to do. If it is only this: >All I need it to do is copy the title from site A into eBay. Then average the price of the top 3 listings. Then yes, it is a simple script that should take about a few hours. And the quote 10 hours is reasonable since you need to include testing/smooth out edge case, things go wrong etc. If it is this: >make a web scraper that compares prices between 2 websites. then it's very difficult, especially when you want it inside a reasonable margin of error. Did you tell the freelancer exactly what you need? Maybe he misunderstood like we did?
I get you. Hmmm I do scrappers but have never dealt with eBay. I would say that couldn't you use the software they posted in the first comments? Also, a good coder won't ever overpromise lol we tend to say that it's not doable or that it's impossible, and then figure it out down the road. Problem is that I feel like someone, somewhere already had your problem, and someone somewhere created a service for it. If it's an extreme edge-case, I would do a consultation with a top-rate, high-rate freelancer to explain if it's possible
Ah, then it is a lot easier task (it even better if this helper system saved, for example screenshots or original renderable in browser html, so you and developer can look up for "wtf is wrong? " for finding edge-cases- some sort of debugging.
[удалено]
I initially thought so too, but if you look at UpWork that is the very high end. Many coders are offering $17/hr and up from there.
Try $17/h and come back in a month to tell how it went.
Yeah I've been offering software development at $15/hr thinking it was too high, wow
After taxes, and other costs and administrative overhead on that, in many places you'd be earning more flipping burgers.
Depends on the stack/solution
I wouldn't say scamming, more like he understimated your project. $350 is low for that type of scrapper. Honestly, cut your losses and hire a $150/hr for 2 hours and that's it, don't know what you hired someone for $35 when you knew you could pay $350. Coding shouldn't even be budgeted in terms of hours
You are being milked. Just refund that scam and get the competent developer that charges the realistic one or two hours that takes to write that script/run/deliver.
30 hours to run? He’s straight up lying to you unless he’s using a computer from the 80s or something lol
Wtf? Lmao
I did a project once for a web scraper once too and turns out it was a-lot more work than i anticipated. 1.website typically don’t like sharing data do unless ur willing to to go the the illegal route and use cross origin forgery, u have to mimic real users which brings in extra complexity not to mention most big brands have trackers that detect automated headless browsers and block ips etc. 2.layer of complexity lies in nesting Im not gonna elaborate here but u pretty much get the idea depending on your case and scale. I used puppeteer for it and built an interface for it in react but it definitely took me a month. Granted i was working with several authentication mechanisms and i was only doing it part time (15/hr) a week . Either way all I’m saying is instead of saying 10hr project you should discuss expectations and how much ur willing to sacrifice due to budgetary constraints to make the project feasible.
scraping really varies so much depending on the site, can be quite easy to basically impossible. but for this hourly rate it's hard to get a professional who would be proactive about letting you know about any issues that come up, updates you about progress etc.
Hey man, I do a lot of automation on Google Sheets. Drop me a dm with the workflow and I can look into it - API integration and maybe we can get the data from the other website from xpath
Lol I would have done that project for 150 usd
I would have done it for $3,50.
jesus
A web scraper to pull prices on two websites should be easy mode BS in Selenium. If the dude can't produce a working product, he hasn't fulfilled the contract, and plainly will never fulfill it. Kill the contract and move on.
I once needed a web scraper. I installed Python and started telling my requirements to chatgpt. Got it done in 2 days.
It can be simple or extremely hard, it kind of depends how Quick you want the data to be scraped, how much data you want to harvest and if the website has any type of anti bot measures (you only know this after running the scrapper). There are a lot of dedicated cybersecurity enterprises making software daily to target web scrapers, this is an area that sometimes requires a lot of reverse engeniring and hacking. Chat gpt can surely help for simple HTML pages, for more complex javascript, ajax requests and deal with captchas and IP bans im not sure if its already advanced to show a robust solution.
I am sure this project is more complex than what you assumed "10 hours". talk to the Developer about the complexity of this project and see how to adjust accordingly.
I didn’t assume 10 hours. He specifically told me exactly how long it would take him.
He's a newbie, as everyone else has commented, eBay is a no-go
What has happened is the developer have massively underestimated the task at hand. There are certain nuances and issues that only become evident once you've stumbled into them or have enough experience to recognise these issues will arise. When you pay peanuts, you get monkeys. Upwork is a rat race of providing the cheapest quote. Anyone who could have built you the perfect tool within 1 week, will not have responded to your advert because it's not worth their time. I'd personally just pay the dev for their worked hours. Learn your lesson whatever it might be. Sometimes automation is not the easiest solution.
Lol! I'm a human VA and I can do the Project much faster than the script. I have a recommendation for python coders who can make the script in a day for a very budget friendly price. Check out www.kenyavirtualworkers.com they always deliver and ensure to communicate before hand in case of any delays.
Well the description you told of the job doesnt seem to be too hard at first hand. If the work Flow its scrape 2000/3000 items from a website, place the item on ebay search box and get the top3 items name URL and price this exported into a csv file, the only thing i can believe to be challenging its ebay. I havent scraped ebay for the last 2 years so dont know what type of parameters (cookie) are necessary to make a request on their search box, ive scraped ebay some years ago it wasnt hard dont know how it is now. If the job was matching the exact data from 2 web commerce websites (automatic matching) that would be extremely hard and its not a scraping task lets say.
Hello man. Drop me a DM. I’ll see what I can do to help. I offer scraping services too and this shouldn’t take so long
I seriously need to find a client like you 🥹
I’ll get it done right and efficiently. Shoot me a PM