• Most Popular
• Blogs
• Subscribe
• Contact Staff

Facebook Makes Major PHP Push With HipHop Windows Veteran Mike Nash Leaving Microsoft Mozilla Releases Firefox 3.5.7 Twitter Users at Risk From Flaw, Researcher Says Google Issues Nexus One Software Update Mozilla Finds Malware in Firefox Add-Ons Apple's iPhone OS Upgrade Has a Security Angle Windows 7 on the iPad? There's an App for That Low-Cost iPad Parts Could Help Apple Cash In

Xbox 360 tests = FAIL Rush to publication Skype Founder Shakes Money Tree For Seesmic SCO is BACK !?!?! ' Linux Next ' Begins To Take Shape Yahoo Moves Ahead With Layoffs Three Hours Of No Email: Priceless Encyclopedia Brown and the Case of the Missing Patch Red Hat Accelerates JBoss Mozilla's New Look Firefox 3 Beta 3   [ More ]

Select a newsletter and click Join to sign up! Newslinx Daily Internet Daily InternetNews Business Report Boston News DC News NY News SiliconValley News InternetNews.com Podcast InternetNews.com Channel XML/RSS Feeds InternetNews.com IT Business News Developer News Ecommerce News Software IT News Hardware IT News Government News Search News Enterprise IT Planet News Networking News IT Security News Storage News Mobility News Web Content News Internet Stats Linux Today

Partner With Us

robots.txt Last modified: Wednesday, May 06, 2009
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.		Related Categories Search Engines World Wide Web Related Terms search engine search engine marketing search results SEO SERP Web Web page Web site robots dot it Products : Top Store Offers: Amazon Marketplace  19.99The Tech Head is a curious being. Voraciously seeking out the latest and shiniest gizmo to add to its heavily laden utility belt, the Tech Head is als... Amazon Marketplace  19.99The Tech Head is a curious being. Voraciously seeking out the latest and shiniest gizmo to add to its heavily laden utility belt, the Tech Head is als... Amazon Marketplace  20.99The Tech Head is a curious being. Voraciously seeking out the latest and shiniest gizmo to add to its heavily laden utility belt, the Tech Head is als... Compare all matches: 3 Store Offers
Webopedia's "Did You Know...?" Section Use this Webopedia knowledge section for an in-depth overview of specific technologies and occurrences in the areas of Computer Science, The Internet, and Computer Hardware and Software. Webopedia's Quick Reference Section Use this Webopedia reference section for information on common Internet and computer facts and occurrences.