How To Block Fasterfox Requests - February 15, 2006
Over the last few months the amount of visitors using Mozilla Firefox has grown to about 25%. The amount of bandwidth being used has also increased a large amount. Part of the reason behind this is that many Firefox users use an extension called “Fasterfox”. This extension “pre-fetches” links on a page so that if the user were to click on a link it would load much faster because its already been downloaded. This may be more convenient for viewer, but is a major problem for many webmasters who are low on bandwidth. Since Fasterfox constantly requests new files, it can cause many servers to overload much faster than if a person viewing the same content without Fasterfox were to view it.
Fasterfox is in fact one of the most popular extensions for Firefox. It is currently ranked as the 3rd most downloaded extension for Firefox on Mozilla Update page. (aka. Firefox Add-ons) The latest version of Fasterfox, v1.0.3, checks the for the robots.txt file on the site the viewer is visiting to check whether it should pre-fetch or not. This new feature allows webmasters to add the following text (in bold) to their robots.txt file to prevent Fasterfox from pre-fetching links. Text To Add To “robots.txt”:
- User-agent: Fasterfox
- Disallow: /
Adding these two lines somewhere in your robots.txt file and placing it in the root folder will prevent Fasterfox from pre-fetching links anywhere on your site. (ex: yourwebsite.com/robots.txt) Webmasters can modify the text so that Fasterfox will only be prevented from pre-fetching on specified directories. Still have questions? Ask us! (reply in the comments)
Links: Fasterfox | Firefox Add-ons
Related: “robots.txt” Tutorial
Recent: Firefox Extend Contest Finalists
posted by Sahas Katta in Tips & Tweaks
on 2/15/2006 at 10:56 AM with Comments (49) & TrackPings (13)
Search
Recent Posts
- June 2008 - Wii/PS3/Xbox Games
- Dual Screen Wallpaper In Vista
- Aliph Releases Jawbone NoiseAssassin
- Wicked Lasers Nexus 95mW (Review)
- LG VX9100 enV2 - Verizon Wireless
- May 2008 - Wii/PS3/360 Games
- Nokia N95 8GB Review
- Setup Multiple Monitors In Vista
- Microsoft Wireless Laser Mouse 7000
- Skatter Tech: Get Your Own Gravatar
Categories
- Uncategorized
- News
- Site News
- Software
- Tips & Tweaks
- Deals
- Podcasts
- Hacking
- Reviews
- Business
- Entertainment
- Hardware
- Gaming
- Autos
- Mobile
- Internet
- Accessories
Archives
- June 2008
- May 2008
- April 2008
- March 2008
- February 2008
- January 2008
- December 2007
- November 2007
- October 2007
- September 2007
- August 2007
- May 2007
Resources
Subscribe
Write a Comment.
49 comments.
Pages: « 1 [2] Show All
Daniel says
if most webhosts use this robots trick, then the extension becomes useless. i download this extension to decrease load times on pages on the web, not just the ‘high bandwidth’ pages.
sahas says
shawgo, your setup should technically work, because all standard search engines would read that and know not to index anything. But the creators of Fasterfox have specified the method posted in this article, therefore if you are really worried I’d just place another two lines along with what you have to once again specify a command for Fasterfox.
Daniel, I don’t think you understood the purpose of this article. It is not to help people bring servers down, but is to help everyone. Please upgrade to 1.0.3 once again. This will help the sites you like stay up. If you do use this on severs that are already over loaded, this will just help bring them down.
Ryan Williams says
I don’t think Daniel understands that webmasters generally pay for limited bandwidth resources, and that in a lot of cases this limit isn’t awfully high - especially on blog websites and such. If one of these websites were to post a good article and have lots of visitors come to read it, there is a significant risk that the site would be knocked offline for the remainder of the month unless the webmaster foots the bill for additional resources. If it’s a hobby site, this is unlikely.
This risk is basically quadrupled (or more) on a per-user basis for everybody using the unfixed extension. So, Daniel, if you’re happy with sites you visit being knocked offline, then sure, go ahead and spread the good word about 1.0.0 ignoring robots.txt. You will then see your entire browser become useless. ;)
Daniel Shawgo says
I added the extra line, just to be safe. Thanks for the advice, Sahas.
Chris says
Daniel:
fasterfox won’t become useless, just restricted. If it becomes annoying enough to content providers, the backlash would make firefox less popular, and firefox themselves would move to block it. controlled is better for everyone. Besides, like the poster above mentions, most of the world will never utilize this.
Personally, I think the prefetch aspect is dumb anyways. I am never interested in 100% of the links on any given page. I use fasterfox but turned off prefetch.
Zach says
Does Fasterfox send an X-Moz header with its prefetch requests? I blocked that back when Google Web Accelerator came out, would it also stop Fasterfox prefetches?
Fred says
The sad part is that Fasterfox is open source, much like every other extension and the mozilla browser itself, which means that any user can just remove the useragent header from the fasterfox extension and totally bypass the robots.txt file.
Cd-MaN says
It would have been nice to say that the developers! of the extension have this information publicly posted:
Admin: Link has been removed because of some part of the URL that corrputed the page.
Here is the new URL: http://tinyurl.com/zctne
Brian G says
User-agent: *
Disallow: /
Should NEVER be done unless you want to take yourself out of all the search engines…. it’s search engine ranking suicide.
Doug says
Not everyone depends on search engines for traffic.
Thanks for the heads up. Its getting ridiculous the number of ways Firefox is being extended (even in the trunk build) to sap bandwidth from websites to give the user a false impression of speed. If only Firefox itself supported robots.txt for its own href prefetching.
John Cox says
Fred: If there is one thing I have learned in all my years of Open Source development, is the overwhelming majority of folks are lazy. I don’t see that to be a problem;
Thanks for the tip!
Ryan Williams says
If the problem becomes seriousness enough than scripts which block what are clearly prefetching from a site for a temporary duration are likely to be be developed - it could even perhaps be set to ignore search engines, using the IP database of something else out there as a headstart.
If people like Daniel insist on sapping people’s bandwidth, the chances are that it will get to the extent were webmasters retaliate with more than casual blocking.
sam says
See more extensions for power surfing.
http://pchere.blogspot.com/2005/12/50-best-firefox-extensions-for-power.html
Name (required) says
> Stwo says
> fasterfox isnt all that anyway… use google web accelerator
While I agree Fasterfox aint all that perhaps you should open your eyes to the fact that not everyone can meet this requirement :
To use Google Web Accelerator, your computer must have a Windows XP or Windows 2000 SP 3+ operating system. Google Web Accelerator works for the Internet Explorer 5.5+ or Firefox 1.0+ browsers.
plus I have 100+ links on quite a few of my pages
Fasterfox is virtually no help to anyone coming to my website
JC says
Yeah, but, at least in the version I have, prefetching is turned off by default…
matt says
Hey Daniel
Thanks for that like to fasterfox 1.0.0
get it here:
http://downloads.mozdev.org/fasterfox/fasterfox-1.0.0-fx.xpi
Update: BUGGER it doesn’t work with firefox 1.5.0.1
Fortezza says
The Fasterfox just forked, so now you can either stick with the original Fasterfox branch, or get FastestFox which does not include the robot.txt check.
Jordan says
Daniel, Matt: Why is it so critical to leave prefetch on? Are you on network connections with enough latency that you really need that enabled for a fast experience? Besides, if you disable the check, webmasters may just decide that they’ll add absolute throttling or write behavioral plugins (easy enough to do — I’m no programmer and even I could do it) to block pre-fetching or ban hosts using it. In fact, if enough folks like you are insistant enough to ignore it when people have politely asked you to stop, I fully expect that to happen. If they’ve asked you to stop, why not play nice and just stop? It doesn’t seem like it’s too much to ask for.
ebauche says
Fortunately my website isn’t really large enough for prefetching to cause too much of an issue (I think) - but I know if people rammed my bandwidth, and I went offline, I don’t think I could easily sort out the cash to pay for more. Ideally I would prefer my bandwidth to be used by people downloading my music than prefetching!
The idea that bandwidth could be used by someone who might not visit those prefetched pages seems crazy.
Cheyne says
Very handy, fasterfox is bad news for us web developers.
http://www.thewebdesignblog.com
Jérémie says
At last, it reads robots.txt. Well, that was much needed.
Ajay D'Souza says
Now that is really good news.
Have added it to my robots.txt Now wait and monitor :)
Any clue on how to check how much bandwidth was it actually eating up?
J says
My websites are update, but most won’t.
Knife In Your Eye says
This is another invention to save our beloved cyberspace… much like segregating nodes via ip address (for the elite), p2p “traffic shaping” (censorship) and use of DRM to (jacking the consumers some more)… Why don’t you twats write a pluggin that will make http requests more effiecent rather then making some crap that limits what people can do… You make the users to be the enemies when in fact the bandwidth providers are the bastards…
Robin says
Thats great that it is finally possible to block prefetching. Before that people could download gigabytes of data that would never be read. As for Daniel and Matt, I ban that type of person from ever entering my websites
Tim says
Prefetch makes web browsing 10x faster easily! I use 1.0.3 & respect those sites that do not want prefetch, but I have never noticed any problems with my own site. (Than again, i’m on University webspace)
robert says
Unfortunately, in some of my tests the robots.txt entry doesn’t stop it…
Daniel says
Good, at least they can bork the robots.txt thing :D
Martin says
Thanks for the info. I was just looking up fasterfox to see if it was worth using. But after reading this i think i will wait the extra half a second to download pages rather than cause problems for some people who spend a lot of their own personal time for the benifit of others. I only checked this out because firefox put a block on the download, maybe they are be responsible.
Mark says
Do you know how to go about discovering installed instances of this plug-in? As in, what if I have some users who have installed this and I don’t want them using it… I didn’t see any additional processes or executables caused by its installation.
Jason Short says
Well, if people are going to ignore the request anyway then webmasters will have to take more drastic measures. We implemented a system where if you grab more than x pages in a 5 second period you are banned. Stopped all that fasterfox traffic. Shouldn’t have to do that. Fasterfox is broken. We had some Firefox users downloading the same pages HUNDREDS of times. We had some users machines that would download 1500 pages in a 5 minute period because of fasterfox… That is just dumb. There is NO WAY a human could view all of them.
David says
The old fasterfox xpi (1.0.0) does work with Firefox 2.0.0.4 if you alter the archive with WinRAR to change the install.rdf version checking.
SiteSucker is still around if you really want everything all at once and don’t care how much damage you do. It’s all about personal responsibility.
Turbo Dude says
I just bumped Fasterfox 1.0.0 to work with Firefox 2.
Now ignoring robot.txt. BWAHAHHAHA!
Anonymous says
The user can easily switch off prefetching. Go to Add-ons in the Tools menu, click Options in the Fasterfox entry in Extentions, switch to Courteous and then back to Custom in the Presets menu, and that is the right place to do this: In the Pipelining menu, Enable Pipelining and Enable Proxy Pipelining, and set Max Pipelining Requests to 10. In Rendering, set Pages In Memory to 10. This procedure should get rid of any prefetching problems and still allow great speed.
Tooez Tooworkaround says
It is too bad that the XPI’s code has released to the general population, now it is a simple matter for anyone with a smidgen of programming knowledge to remove the polite-man check and one rouge group called something like Frozey Crew 200* has released a version that is simply nastier than anything that has come before it that utilizes the entire available bandwidth and when it runs out it seeks other sites that have been somehow selected at ramdom to continue the data pull. Just friggen nasty…Why people have to do crud like that is beyond me, there out to be international laws against those kind of nasty-hacker groups.
Dwayne Charrington says
I just implemented this onto my website. That’ll teach those bandwidth thieves a lesson or two. Also might lower the “Digg Effect” considering that a lot of people who visit from Digg are probably using this extension. When your server is taking a beating from 100,000 people and 20,000 of them are using Fasterfox without this your server is already gone.
- Dwayne Charrington.
http://www.dwaynecharrington.com
jenny says
what a hell are u talking about ???
I didn’t figure it out!!!!!
is it dangeros ??
Jacky says
Since there is a FasterFox fork which ignores the robot.txt I will implement a blocking mechanisim. Care needs to be taken not to block search engines but I think this can be done easily.
- Tell all search engines not to spider a specific directory via robots.txt
- Setup a hidden link on every page, let’s call it VandalTrap in this example
- Run a server side script such as PHP on the index page of the VandalTrap directory. If the page is loaded, the .htaccess file is modified to block the user
This could also be extended to display a informing message but if they choose to install software which circumvents the rules then blocking the IP for 20 minutes should be the best choice.
Pages: « 1 [2] Show All
13 trackback / pingbacks:
dotCOM addendums » How To Block Fasterfox Requests (February 17, 2006)techactually » Blog Archive » Blocking Fasterfox requests (February 17, 2006)
WTFDYWMTDAI (February 17, 2006)
The.RSS.Reporter (February 18, 2006)
MacTheWeb » Blog Archive » Fighting Fasterfox (February 18, 2006)
raptxt.it » Come bloccare il pre-fetching di Fasterfox (February 18, 2006)
Creation Robot (February 18, 2006)
The Daily Thing Blog - Blocking Fasterfox To Save Your Sanity and Your Wallet (February 18, 2006)
padawan (February 19, 2006)
Snakey’s WeBlog » Blog Archive » How to block FasterFox (February 22, 2006)
19-03-06 - BRIGADOON………2006 (March 19, 2006)
Programming » How To Block Fasterfox To Save Bandwidth (August 23, 2006)
7 Green-themed/Earth-friendly Sites #2 » BoltPress (June 20, 2008)
Leave a Comment