• Most Popular
• Blogs
• Subscribe
• Contact Staff

Droid Upgrade to Android 2.1 on Hold Nexus One: Signs Point to Verizon Debut Soon Facebook Users Targeted in E-mail Scheme Open Source Developers Pick Android Over iPhone Droid OS Upgrade Could Drop This Week Nexus One: Failure or a Late Bloomer? Bing Continues to Gain Share SAP CEOs Share Plans for SaaS, Battling Oracle EMC Lays Out Global Storage Vision Intel, AMD Seen Headed for Lift in Q1 Earnings

Xbox 360 tests = FAIL Rush to publication Skype Founder Shakes Money Tree For Seesmic SCO is BACK !?!?! ' Linux Next ' Begins To Take Shape Yahoo Moves Ahead With Layoffs Three Hours Of No Email: Priceless Encyclopedia Brown and the Case of the Missing Patch Red Hat Accelerates JBoss Mozilla's New Look Firefox 3 Beta 3   [ More ]

Select a newsletter and click Join to sign up! Newslinx Daily Internet Daily InternetNews Business Report Boston News DC News NY News SiliconValley News InternetNews.com Podcast InternetNews.com Channel XML/RSS Feeds InternetNews.com IT Business News Developer News Ecommerce News Software IT News Hardware IT News Government News Search News Enterprise IT Planet News Networking News IT Security News Storage News Mobility News Web Content News Internet Stats Linux Today

Partner With Us

robots.txt Last modified: Wednesday, May 06, 2009
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.		Related Categories Search Engines World Wide Web Related Terms search engine search engine marketing search results SEO SERP Web Web page Web site Shopping robot dot Products Compare Products,Prices and Stores Shop by Category: Sport and Outdoor 23 Store Offers Toys 2 Store Offers
Webopedia's "Did You Know...?" Section Use this Webopedia knowledge section for an in-depth overview of specific technologies and occurrences in the areas of Computer Science, The Internet, and Computer Hardware and Software. Webopedia's Quick Reference Section Use this Webopedia reference section for information on common Internet and computer facts and occurrences.