[CALUG] robots.txt

[Walt Smith]

Hi,

I have an account over at lonestar.
I've seen tidbits about a file used apparently
for search engines called robots.txt.  I don't know
in what "domain" it's used exactly.

Just wondering if I could get a short synopsis
from someone about whether that file is used as 
a site file, or whether a user can use it to 
"encourage" searches at his web page?  i.e.
have their own robots.txt file.   OF course I 
would assume a site file would have a narrowing precedence
over a user account... ( can I add a robot.txt file
for a personal web page in a  *nix style system?)
this is all assuming I actually have some idea
what the file is used for.

also, in the html web page file, are keywords put into 
some html top section <meta something ??> still used and
is it really useful for search engines ?

I'm not so much interested in overtly trying to push or advertise,
but I don't want to discourage a search engine for a particular
web page; i.e. to let them find something.

thx,

Walt......