Mirror Remote Files With Perl and PHP
The index page of one of the sites I maintain dynamically includes a small html page sourced on another server. Sometimes accessing the other server caused delayed appearance of my page. I decided that if the local index page used a mirrored version of the included remote html content, it would always appear quickly. The trick would be to script and schedule a regular update of the local version of the included content file.
I show two mirror technics here, Perl and PHP.
A Google-search supplied a link to http://www.perl.com/newdocs/pod/perlipc.html#Sockets_Client_Server_Communica where there was a big help script for internet TCP clients and servers. I added my host and file specifications in lines 8 and 10.
The file I wanted to retrieve had a header returned by the remote server and an image reference to their logo. The address for the image file was, of course on the remote server. The fetched file also had a non-html server prologue, so I added code to save only html lines (those containing "<") and replace the remote path with a local "images" path (line 25): (NOTE: I use images here because some characters like the backslash "" are not treated well in browsers and content managers.)
The next step was to schedule the update using cron on the local server. I created a cron specification file with
echo "0,20,40 * * * * perl /home/sites/site64/cgi-bin/mir.pl" > /home/sites/site64/cgi-bin/mirsched
and then issued the command to execute the "mirsched" file as a crontab entry
Lo, and behold, "lthead.html" is captured, cleaned and localized three times an hour.
Here‚?ā¨∆?ĄĘs another mirror script that grabs the daily bikini image from http://www.thedaily.com/bikini.html/
This is a very adaptable script; here‚?ā¨∆?ĄĘs yet another version that hunts down Yahoo!‚?ā¨∆?ĄĘs "top stories" headlines:
Here's yet another version in PHP that saves the top 10 Yahoo Top Stories in the news into a local file. You can then include the file into a web page with an SSI include statement.
Install the script in your web /cgi-bin/ directory, then schedule the script to run using "crontab -e" command and enter the line:
00 * * * * php /var/www/cgi-bin/mir-yahoots-file.php