Home > Apache, cakePHP, CSS, HTML, IE, JavaScript, PHP, Zend > PHP seo sitemap spider robots.txt robots allow disallow search engine nice looking url

PHP seo sitemap spider robots.txt robots allow disallow search engine nice looking url

Introduction:

First of all robots.txt file must, to allow a bot (spiders like google, yahoo, msn etc) to reach your webiste and crawl your:  “Damn looking url” or “Ugly looking url!!” or “Nice looking url”, what ever you name, just name it for standard understanding we name it SEO.

ex: http://www.site.com/robots.txt:

User-agent: *
Disallow: /css/
Disallow: test.php/
Allow: /file/myfile.html
Sitemap: /sitemap/xml.xml
Sitemap: /sitemap/txtmode.txt

Other methods, way of doing:

1. Text based (KISS, keep it simple STUPID!!!)  && 10mb && 50,000 lines

a. http://mysite/searchlist1.txt:

http://www.mysite.com/index.php?a=b=c=d=e=f=g=h=j=uglylooking_url
http://www.mysite.com/index.php?a=b=c=d=e=f=g=h=j=damnlooking_url
http://www.mysite.com/hello-world-whats-up

b. http://mysite/searchlist2.txt

 http://www.mysite.com/index.php?a=b=c=d=e=f=g=h=j=uglylooking_url
 http://www.mysite.com/index.php?a=b=c=d=e=f=g=h=j=damnlooking_url
 http://www.mysite.com/hello-world-whats-up
 

c. http://mystei/robots.txt

....
Sitemap: http://mysite/searchlist1.txt
Sitemap: http://mysite/searchlist2.txt

2. Xml based (Not KISS)

3. Meta tag/title tag (KISS)

You are upset/World is so crud:
Nice looking url all of a certain, became the bible for web technologies, and you don’t have any solution. Wait! try this atleast:

vi /etc/httpd/conf/httpd.conf
Alias /my-nice-urls http://mysite/index.php?a=old=b=nasty=c=ugly

Before:
your bad url  was: http://mysite/index.php?a=old=b=nasty=c=ugly
After:
your working url is: http://mysite/my-nice-urls

More reading:

http://en.wikipedia.org/wiki/Sitemaps
http://en.wikipedia.org/wiki/Robots_exclusion_standard
http://www.sitemaps.org/protocol.php

Advertisements
Categories: Apache, cakePHP, CSS, HTML, IE, JavaScript, PHP, Zend
  1. May 21, 2010 at 9:21 pm

    Thanks good article.

  2. October 20, 2010 at 10:02 am

    I think most SEO techniques are a waste of time. Nothing beats good content and genuine backlinks

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: