Running a web spidering service - potential legal problems?

Discussions
#1

Hi everyone!

There are many services out there which scan the web for you. For example, some notify you if your product gets mentioned in social media.
I had a similar business idea. It would involve downloading quite a few websites and analyzing their content, and I would offer that as a web app.

However, I’m concerned about the legal aspects. My idea would not violate copyright. But it could easily happen that my service accidentially downloads illegal websites that match some search terms.

Well-funded startups might have the resources to defend against that, however as a solo bootstrapper it sounds like a dangerous undertaking.
Has anyone tried something like that, or has some opinions on that issue?

#2

I Am Not A Lawyer, but this seems like a good time for you to consult one… :smile:

I was looking at an app that would have required screen-scraping, only to discover that many of the sites I would be targeting expressly forbade it in their Terms Of Service. Have you looked at the TOS for websites you’d be targeting? (Or are you targeting any random websites the user specifies?)

On the other hand, all the search engines rely on spidering. Maybe if you honor the websites’ robots.txt file?

  • Jack
#3

What do you call an “illegal website”? Guns, drugs and child porn?

As already said, ToC seems a more common concern… but then the site owners wouldn’t waste time fighting you if you only visit them once in a while, generating a little load. And if they send you a cease and desist, well, just comply.