You should be able to get the desired affect using django.http.HttpResponse so long as you specify a MIME type, ala:

from django.http import HttpResponse

def robots(request):
    return HttpResponse(open('robots.txt').read(), 'text/plain')

I imagine you could also do it in httpd.conf using the FilesMatch directive.

Actually, you were correct.

It's strange but sometimes, even though I do a stop and start, it take a few minutes to see the results of a change.  I was still seeing the results of what I had done before I tried your code.

Thanks for the help David!

-Nino

Nino wrote:

Does anyone have an example of using their Robots.txt file under Django that they would like to share?

I tried to just make a static app, but I'm already up to my allowed 3 apps.

You can serve static files from your own Apache. I have these lines in my httpd.conf:

# robots.txt
<Location "/robots.txt">
    SetHandler None
</Location>

alias /robots.txt /home2/blogcosm/webapps/django/htdocs/robots.txt

There were three problems:

1) You didn't restart Apache after changing httpd.conf.
2)

[mcf2008@web28 django]$ ./apache2/bin/stop
Stopped
[mcf2008@web28 django]$ ./apache2/bin/start
Syntax error on line 30 of /home/mcf2008/webapps/django/apache2/conf/httpd.conf:
Invalid command 'alias', perhaps misspelled or defined by a module not included in the server configuration
Restarted
[mcf2008@web28 django]$

3)

[mcf2008@web28 django]$ ls /home2/mcf2008/webapps/media/robots.txt
ls: /home2/mcf2008/webapps/media/robots.txt: No such file or directory
[mcf2008@web28 django]$

So I:

1) Added "LoadModule alias_module modules/mod_alias.so" to your httpd.conf.
2) Created robots.txt.

And now everything appears to be working as expected.

Hope it helps.

Last edited by IAIHMB (2008-02-23 09:50:17)

Hi Lanec,

It looks like you figured it out.  I can see your favicon of the little house when I visit your site.  Would you mind posting the solution?

Regards,

~Chris Sebastian
(admin)

If you check your error log, you'll see the problem:

.htaccess: Invalid command 'Header', perhaps misspelled or defined by a module not included in the server configuration

I added the header module to your httpd.conf and now it works

DiegoCerdan wrote:

@likebike: I am doing the same for favicon as for robots and seems its working. If you try to check it through Firefox you get crappy output because mime type is not set.

I spent some time trying to get favicon working and finally realized that, at least for my setup, I had to have the favicon location and alias *below* the root django location directive.  Then to get the mime type correct in Firefox, add "ForceType image/vnd.microsoft.icon" to the favicon location.

Last edited by dburke (2009-06-23 15:16:38)

I'm having a problem trying to do use robots.txt and favicon.ico on my Django site.

I have created robots.txt and favicon.ico and added them to my Media App (latter in subfolder /icons/)

If I use URL  media.username.com/robots.txt and  media.username.com/icons/favicon.ico   - I can see the text and image as expected. Good.

I updated the Httpd.conf with

<Location "/robots.txt">
    SetHandler None
</Location>
<Location "/favicon.ico">
    SetHandler None
</Location>
alias /robots.txt /home/username/webapps/djangomedia_staticcgi_sym/robots.txt
alias /favicon.ico /home/username/webapps/djangomedia_staticcgi_sym/favicon.ico

So this should redirect the URLs to the media folder to get the two files as appropriate

But I cannot see the favicon with the Website URL and adding robots.txt or icons/favicon.ico to the URL results in a 500. Checking the Apache Error log gives
/home/username/webapps/djangomedia_staticcgi_sym/.htaccess: Invalid command 'ExpiresActive', ...

I use an .htaccess file in the static media folder for ExpiresHeaders

<FilesMatch "\.(ico|pdf|flv|jpg|jpeg|png|gif|js|swf)$">
    ExpiresActive On
    ExpiresDefault "access plus 1 month"
</FilesMatch>

Looking for help there is a suggestion that I require to add the following to my httpd.conf although oddly this was not suggested when I started using ExpiresHeaders and there have been no problems previously.

LoadModule expires_module modules/mod_expires.so
AddModule mod_expires.c

And I have been restarting Apache

With a little more digging I discovered that removal of the .htaccess allows the robots.txt to appear and for the favicon.ico to become visible with www.username.com/favicon.ico although it wasn't appearing as an actual favicon if you follow.

I think the expires_module is a red herring - I didn't need it before to use ExpiresHeaders
In summary, trying to redirect the two files via httpd with an ExpiresHeaders in the .htaccess of my Media causes a 500.


Also, I'm not sure of favicons are picked up automatically or if I also need to add <link rel="icon" href="favicon"....  to the HTML Template. I've added this anyway.

Any ideas?

Last edited by Parthian (2009-09-18 10:22:52)

That's the trouble with finding solutions on forums - they tend not to age well. AddModule mod_expires.c  - is an out of date solution as Seanf explained on a Ticket answer. And I don't need the <link rel="icon"   HTML.  but Temp Files / Cache need to be cleared to see the favicon.

Great support.

Actually, that did it. After I posted that previous comment, I realized that I had some conflicting data in the conf file and sure enough, after I removed it, the favicon showed up.

Thanks!

Hey lazyones like me.
http://bitbucket.org/jezdez/django-robots/src/