Help! How to write .htaccess for Dynamic Mirror

[WebsiteMaven]

Eric (BigBison) has been helping me figure out some ways to optimize on of my websites. He turned me on to this feature in Apache and I'm at a loss at how to get the syntax correct in my .htaccess file:

Dynamic Mirror
Description:
Assume there are nice webpages on remote hosts we want to bring into our namespace. For FTP servers we would use the mirror program which actually maintains an explicit up-to-date copy of the remote data on the local machine. For a webserver we could use the program webcopy which acts similar via HTTP. But both techniques have one major drawback: The local copy is always just as up-to-date as often we run the program. It would be much better if the mirror is not a static one we have to establish explicitly. Instead we want a dynamic mirror with data which gets updated automatically when there is need (updated data on the remote host).

Solution:
To provide this feature we map the remote webpage or even the complete remote webarea to our namespace by the use of the Proxy Throughput feature (flag [P]):

RewriteEngine on
RewriteBase /~quux/
RewriteRule ^hotsheet/(.*)$ http://www.tstimpreso.com/hotsheet/$1 [P]

RewriteEngine on
RewriteBase /~quux/
RewriteRule ^usa-news\.html$ http://www.quux-corp.com/news/index.html [P]

Let's say I want to dynamically mirror webpages from http://www.cooldomain.com into my domain: http://www.mydomain.com. I want it to dynamically mirror from the root and thoughout the site. In other words if I type:

http://www.mydomain.com/page1.html
the page would be displayed on my domain but would be a dynamic mirror of all the content at:
http://www.cooldomain.com/page1.html

I've tried going to the public_html folder of mydomain.com and configured the .htaccess file with the following:


RewriteEngine on
RewriteBase /
RewriteRule ^/(.*)$ http://www.cooldomain.com/$1 [P]

Is the syntax correct?
It seems I have the option, according to the Apache Rewrite Guide (http://httpd.apache.org/docs/2.0/misc/rewriteguide.html), to configure this in Apache but I'm a novice when it comes to server configurations. How would I do that or where else might I look to get this Dynamic Mirror feature to work?

[Anoop]

The following entry in .htaccess is working for me.

RewriteEngine On
RewriteRule (.*) http://www.cooldomain.com/$1 [P]

If it does not work for you, then try adding a redirect and check if it is working.

RewriteEngine on
RewriteRule (.*) http://www.cooldomain.com/$1 [L]

And also make sure that apache is checking the directives in .htaccess. You can check it by placing some garbage values in your .htaccess. Then apache will
show "Internal Server Error".

[WebsiteMaven]

Thanks Anoop!

Still not working with those lines. I confirmed that I got an internal server error when I added garbage.

I should be more specific to help figure this out. I'm trying to get a subdomain to dynamically mirror a 2nd level domain. Here is the full .htaccess for forward.solideogloria.com in the public_html directory. Can you detect anything that might be causing it not to work.:
# -FrontPage-

IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*

<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName www.forward.solideogloria.com
AuthUserFile /home/forward/public_html/_vti_pvt/service.pwd
AuthGroupFile /home/forward/public_html/_vti_pvt/service.grp
RewriteEngine On
RewriteRule (.*) http://www.websitemaven.com/$1 [P]

I get an HTTP 404 Not found error. Any ideas?

[Anoop]

Try adding the rewrite rules in the .htaccess in the document root of your
subdomain. I think it is public_html/forward (If it is a CPanel server)

[Anoop]

Mostly your problem might be due to lack of mod_proxy support in apache.
In order to use the [P] flag, your apache should have support for mod_proxy module. The default CPanel configuration doesn't have mod_proxy support.

Contact your hosting support and check whether the server has the mod_proxy support. If you have the "root" user access to the server, check for the following entry in http.conf.

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so

[Jamie]

Moved to "Programming". :)

[WebsiteMaven]

Anoop (or others),

My server was having problems and I'm just now getting back to this. I was able to go check my httpd.conf file and those lines are missing. Not only that, however, I don't think they installed the mod_proxy modules you mention above.

Can I just create a folder called modules and stick them there and refer to them in my httpd.conf and then restart the server and it should work?

I just want to make sure I'm a bit more certain about the steps before I mess something up.

[sea otter]

Rich, is this the Jag VPS? Which OS and control panel?

Basically, yeah, you'll need to install the proxy modules (I think you'll need to compile them via apxs) and then add those lines to httpd.

You probably already have a module directory: /etc/httpd/modules which is where they should go. You might also need to set some other options as hinted at in this message on the CentOS mailing list:

http://lists.centos.org/pipermail/centos/2006-April/063311.html

To be honest, this is just the kind of thing Jag support has been great at doing for me, but I know you're somewhat leery of relying on them right now.

[WebsiteMaven]

Rich, is this the Jag VPS? Which OS and control panel?

Basically, yeah, you'll need to install the proxy modules (I think you'll need to compile them via apxs) and then add those lines to httpd.

You probably already have a module directory: /etc/httpd/modules which is where they should go. You might also need to set some other options as hinted at in this message on the CentOS mailing list:

http://lists.centos.org/pipermail/centos/2006-April/063311.html

To be honest, this is just the kind of thing Jag support has been great at doing for me, but I know you're somewhat leery of relying on them right now.
Yeah it's the JAG site with CentOS and Cpanel.

I'm warming up to them but I wanted to play around with it a bit to learn a little bit about administering stuff myself. While I'm "playing" though, I don't want to wreak havoc. I'll look again but I didn't see a modules directory under httpd.

[sea otter]

Ah --I'm not sure how "borglike" CPanel is with its tendrils. I use DirectAdmin, which means I can compile/build anything I want behind the scenes and not have the control panel tromp on it.

You *might* need to build/install the modules (and possibly update apache) via cpanel.

When I used CPanel in the past, I joined their forums and trawled the board for answers. Always found what I needed:

http://forums.cpanel.net/

At worst you'll find nothing, and then we can get back to ssh, wget, rpm and a manual install!

[WebsiteMaven]

I was looking at some documentation and these files are just configuration files. I've got shell access (WinSCP) and I'm going to copy them into a modules directory under httpd and then modify httpd.conf and then start and stop the service and it should work.

[sea otter]

errm...the ".so" files are shared module files -- unix equivalents of .dll files in Windows. They need to be specifically compiled on the platform for which they are intended to be run, using the apache module configuration system "apxs" (which does some configuration stuff and then a normal gcc build).

You might be able to get precompiled versions via rpm or yum.

Also, the mod_proxy docs at apache are here:

http://httpd.apache.org/docs/1.3/mod/mod_proxy.html

They have some examples, and you might need to set more things up in httpd.conf beyond simply loading the modules.

[WebsiteMaven]

I used the ones that were in the /usr/lib/httpd/modules directory and copied them over into the /etc/httpd/modules directory.

I tried adding the lines to httpd.conf above (backed up current version first). I restarted the http service and it failed to load. I changed the httpd.conf back to it original verson and got rid of the modules directory. http started up fine.

I contacted tech support and am going to have them do it. ;)

Thanks for that link.

[sea otter]

oh ok, you already had 'em. After you copied them, the configuration lines needed to be changed to reflect /etc/httpd/modules as the path to the libraries. Don't know if you did that.

Regardless, let me know how long it takes them, and if they set it up correctly!

[WebsiteMaven]

They got back to me right away. They said the following:

Hi Rich,

Sorry, running proxy servers or proxy sites/scripts on your vps is against our AUP.

Regards,
Veena
Is Dynamic Mirror considered a proxy server or proxy site?

[sea otter]

Oh wow. No, it isn't. Are we certain that using the [P] flag is the only way to get mod_rewrite to do this?

I always start at http://www.modrewrite.com/ when looking for stuff like this. The forums are good, as is the sometimes-confusing Ralf Engelschall guide ('tho on first glance it looks like Bison's original example might have come from there).

EDIT: Just looked -- yeah, Dynamic Mirror is right out of RE's guide.

Maybe PM Jag and explain? He might make an exception given who you are and what the sites involved are.

[WebsiteMaven]

Wasn't even necessary. They ended up doing it after I told them exactly what I was going to use it for.

[sea otter]

Excellent. Glad it worked out.

[BigBison]

EDIT: Just looked -- yeah, Dynamic Mirror is right out of RE's guide.

Yup, old school, that's me. :D I haven't configured an Apache server since the late 90's, though.

[sea otter]

I recognized the "quux" part. Ahhh, the good old days of foo and bar and baz. Old-Skool is fine by me; I've been using Slackware and FreeBSD since back in the floppy days.