Keith's Software and Tutorials Home Page
Knowledge Is Power
CGI Banner




Sections on
this page:





Getting Started
A lot of people have Web pages but most feel that CGI scripts are "over their head". Nonsense! If you know basic HTML and know how to use an FTP program like WS_FTP to transfer files, chances are you can be using a CGI script on your Web pages in about 15 minutes. With so many free CGI scripts available (including Bestdam Logger Lite) you are really short-changing yourself if you are not using the CGI capabilities offered by your ISP or host provider to use CGI scripts.

While there are a lot of "CGI Tutorial" pages out there, most deal with how to write CGI scripts. For those who just want to know how to use CGI scripts, information is pretty scarce. That's why this page was created.

What Is CGI ?

There are a number of different methods Web developers can use to enhance the content of Web pages over and above what simple HTML provides. Most of these methods involve writing little programs or routines using one scripting language or another. The are two basic differences with these methods:

  1. Where is the script code located ?
  2. Where is the script code executed ?

The following table summarizes to the above two points for the various methods:

Method Where is the script
code located ? 		Where is the script
code executed ? 		Required HTML
file extensions
CGI In files in the CGI-BIN
directory on the server. 		On the server. .shtm or .shtml
PHP, ColdFusion, ASP Embedded in the
HTML document. 		On the server. .php .cfm .asp
Javascript Embedded in the
HTML document. 		On the user's PC
by their browser. 		n/a
Java * In files on
the server. 		On the user's PC
by their browser. 		n/a

* Note: Java is not a scripting language. Java files are pre-compiled "applets". The applet files are stored on the server and downloaded by the browser for execution.
Even though the script code for PHP, ColdFusion, and ASP is embedded in the HTML code, it's not visible at the browser (when using View/Source). Before sending the page to the browser, the server strips out the script code, executes it, and puts in its' place the results of executing that code. For example, a script command to return the current date will be stripped out and the text of the current date will be put in its' place and in the HTML that's sent to the browser. As such, the location of the script code embedded in the HTML is the position of the execution results on the Web page.

The process is somewhat the same with CGI scripts. CGI is utilized by placing an appropriate HTML tag (called an SSI directive tag) in your HTML code. (The author of the script you wish to use should provide you with the appropriate HTML tag needed to run that script.) When the page is requested by a browser the server reads the tag (and strips it out), executes the server-located script file that's specified by the tag, and puts in the tag's place the results of the execution of the script file. A common example is a hit counter script. The script execution increments the counter and the text of the resulting count is put in the HTML that's sent to the browser so that it appears on the page in the same place where the SSI diective tag was located.

If you've ever looked at your browser's settings, you probably seen check boxes or radio buttons to enable or disable Javascript and Java but haven't seen anything for CGI or PHP. That's because Javascript and Java are executed by the browser (or not, if you disable them). Your browser doesn't know anything about CGI or PHP. It just gets pure HTML from the server after the scripts are executed.

The embedded script method (PHP, ColdFusion, ASP) is mainly used by developers writing "front end" web pages that will access "back end" databases (i.e. client/server Web applications). The big advantage of the CGI method is that the scripts are stored in files and there are literally thousands of freely available scripts already written and ready for you to download and use on your Web site. This means that you don't have to learn a scripting language in order to get the benefits of scripts. Someone has already done the work for you.

You've undoubtedly visited Web pages and seen "cgi-bin" appear in the location line of your browser. CGI stands for "Common Gateway Interface". When you see that "cgi-bin" appear on the location line, you probably executed a CGI script on the server when you requested the page.

Two of the methods shown in the above table have the code executed by the server. But how does the server know to look for a tag which calls a CGI script or to look for embedded script code in a PHP page? It's done using different extensions when naming HTML files. If a browser requests a page (an HTML file) with a .shtml extension, the Web server knows it should "parse" (i.e. look through) the page for a tag which calls a CGI script and execute that script before sending the page to the browser. If the requested page has a .php extension, it knows to look for and execute any embedded PHP code it finds in the page before sending it to the browser.


CGI and Perl

You will often see the term "Perl" used with the term "CGI". The two are NOT the same. CGI programs, or scripts, can be written in a variety of computer languages, including C. CGI is the process by which scripts are run. Perl is the most common language used for writing CGI scripts, and for very good reason. (See the Messin' Around with Perl section below).

Because Perl has its' roots in UNIX, many people think that Perl CGI scripts cannot be used on Windows NT Web servers. Not true! Perl CGI scripts can not only run on UNIX and Windows servers, but with a little tweaking for "AppleScript", many can run on Macintosh servers as well.


Your CGI

Most ISPs that offer Web space and Website "hosting" companies support the use of CGI scripts. It is so common in fact, that if your ISP or host provider doesn't offer it, you should consider taking your business elsewhere. The two questions that need to be answered are:
1. Do I have the capability of running my own CGI scripts ?

2. Does my CGI capability include support for SSI (Server Side Includes) ?

    Note:  Don't confuse SSI with SSL (the Secure Socket Layer protocol used with browsers), they're two entirely different things.

If the answer to both of these questions is "Yes", you're good to go. You can run most of the scripts available on the Web. There are some scripts that don't require SSI but a lot do so having SSI support will allow you to run more scripts. However, if you don't have it you can still run some scripts. The documentation (readme file) should come with the script and state whether it requires SSI support or not.

"Server-Side Includes" are just that, commands (aka "directives") to the Web server to include some information the server has in the displayed Web page. A common use of SSI is to display the current date and time on a Web page. These commands are enclosed within HTML comment tags (<!--   -->) in a Web page so the browser ignores them. These comment tags with an enclosed server command are the "SSI directive tags" that are mentioned below. When these SSI directive tags are used with scripts, the "...information the server has..." is whatever output was generated by the execution of the script. The script will dictate whether this information is displayed on the Web page (as with a hit counter) or written to some file (as with a logger). As an example, here is the HTML (SSI directive) tag for Bestdam Logger:

<!--#exec cgi="/cgi-bin/bdlogger/bdlogger.pl" -->


Back to your CGI situation. Most Web hosting services and ISP's have a Technical Support section on their Website that may also have a "FAQ" (Frequently Asked Questions) page. Snoop around their support pages and see if they if you can find anything related to CGI and SSI that may answer the above two questions.

If all this talk of Perl, SSI, servers, etc. has you a little confused, here's a diagram to show the inter-relationship of the various components of a typical Internet server. Note that "Apache" and "Sendmail" are like brand names. There are other Web-sever and e-mail-server software packages available.

Server Diagram
Big-bucks setup, right? WRONG! You can download a Debian Linux DVD for free and it includes the Apache (which has CGI and Perl modules) and Sendmail software. Linux will run on an old Pentium with 32 meg. So if you've got an old system collecting dust and a broadband connection to the Internet, for $20. you can have your own Internet server and eliminate the need for a Web hosting service.

There are both UNIX/Linux and Windows versions of Apache. However, most Windows servers use IIS (Internet Information Server) which is included as part of the Windows server software. There is also a freeware Windows server e-mail program called Blat that will allow you to send e-mail using scripts. There is a link to the Blat Website on the Bestdam Logger Setup page.


One way to tell if you have CGI capability is if you have a sub-directory (folder) in your root Web directory called cgi-bin. If you do, you very likely do have CGI capability. The only question then is, does your CGI capability include SSI support ? If you didn't find an answer to this on their Website, you will have to check with your host or ISP.



If you don't have a cgi-bin sub-directory that doesn't necessarily mean you don't have CGI capability. It could mean it hasn't been set up.

Normally you cannot simply create the cgi-bin sub-directory yourself. It is a special sub-directory that must be set up by a system administrator. However, I have seen hosts that allow you to create the sub-directory using an FTP program and use their Website "admin" function to enable it. If you are able to create it, how to CHMOD this sub-directory is given at the end of the Transferring Files and Permissions section below.

Note that your host provider or ISP may set up a sub-directory called simply cgi rather than cgi-bin. This is the same thing and you would just need to make the necessary changes to any tags you add to your Web pages to run scripts.

In some cases, hosts or ISPs that do not offer CGI capability with their base package will offer it as an "add-on" or optional service for an additional fee.


Three Steps To Using a Script

Once you've established that you do have CGI and SSI capability, and you've downloaded the script you want to use, there are three basic steps you need to take in order to use the script on your Website:

  1. Set any options that the script may need

  2. Transfer the script's files (the script file itself and any necessary data files) to your Web server and set the permissions

  3. Add the script's HTML tag to the page(s) you want to use the script
Perl scripts typically have a .PL extension, but they may also have a .CGI extension. (Files with other extensions, or no extensions, will likely be data files used by the script.) "Setting options" in scripts is typically just a matter of opening the .PL (or .CGI) file in a text editor and entering values for some of the scripts variables. For example, you may need to enter your e-mail address if the script sends e-mail notifications of some event. Information relating to Step 1 (setting options in scripts) is covered in general in the next section Using Free Scripts Found On The Web.

Note:  Some hosts or ISPs may require that scripts have a .cgi extension. It is normally not a problem to just rename the file to comply. If you do so, remember to change the extension in the HTML tag also provided in the script's documentation.
Once the script is all set up and ready to go, the next step is to transfer the script files to your Web server and set the proper "permissions" to the files. These permissions are necessary so your Website visitors can access them properly. Step 2 is covered in detail in the Transferring Files and Permissions section below.

That takes care of the script side of things. The final step is to add the appropriate HTML tag to your Web page (HTML file) to call the script and then transferring that updated page to the server. The documentation that came with the script, or the comments in the script file itself, should include the appropriate tag to use. But remember that you may have to modify this tag. Most tags assume your CGI sub-directory is called cgi-bin. If it's called simply cgi, or something else, make the necessary change in the tag. Step 3 is also covered in detail in the Transferring Files and Permissions section below.

The discussion in the next section (Step 1) will be generic so that it applies to most scripts available on the Web. Steps 2 and 3 (transferring files and adding the tag to an HTML page) are covered in greater detail in a later section using the free Lite Edition of Bestdam Logger as an example. However, once you've seen the process in action you can easily apply it to other scripts.

If you're not familiar with loggers, they collect information about those who visit your Website. Bestdam Logger logs date and time, page viewed, visitor IP address, domain and client info, and the page they came from, called the "referrer". This information can be valuable in answering questions such as

  • Who (via what domain) is visiting my site ?
  • Where are they coming from (i.e. who's sending me traffic) ?
  • What are the peak traffic times ?
  • Which pages are the most popular ?
  • Which browser is most often used to view my site ?
  • Which search engines are "spidering" my site ?
  • Which search keywords did visitors use to find my site ?
Knowing search keywords can be helpful in determining which META keywords are effective. If the logger has "multi-page support", and you enable the logging function on all of your pages, you can track the paths visitors took through your site.

Taking the above Internet server diagram and adding another box and a few more lines, you can see how your HTML file completes the process.

Server Process


Top of page



Using Free Scripts
As mentioned previously, most CGI scripts are written using the Perl language. When a Perl programmer writes a script they may choose to make it freely available to everyone on the Web. However, you should use caution when selecting these free scripts. I have seen many instances where these free scripts do not adequately "lock" files (which is important in a Web environment where multiple people could be viewing the same Web page simultaneously). Some poorly written scripts could also actually pose a security risk by allowing unauthorized access to the server. Unfortunately, there is no easy way for the untrained eye to determine if adequate file locking is used or if the script represents a security hole. And most Websites that offer scripts from third parties for download do not perform any sort of quality control checks. When in doubt, ask your host or ISP to look over the script you want to use. Most would much rather take a couple minutes to evaluate a script than clean up the mess a poorly-written script could cause.

The Files

Perl scripts typically have a .PL extension, but I have also seen them with a .CGI extension. Files with other extensions, or no extensions, will likely be data files used by the script. Serious Perl script programmers will also include a readme file that contains information about the script and how to set it up. This readme file is intended for those who will be using the script and does not have to be transferred to the server with the script and data files. Throughout this section I refer to .PL files. However, the same would apply to .CGI files if your script has that extension instead.

More complex scripts may have more than one .PL file which may require different HTML tags for each one. (It's also possible that one script may "call" another script so that only one tag is needed.) Also, be on the lookout for additional .PL files with names like config.pl or cfg.pl. These are script files where all of the user-settable options are entered and stored. If a file like this is included in the download file, you typically don't have to open the main script file to set options. The main script will refer to this configuration script each time it is executed.

All of these files are typically combined into a single .ZIP file for you to download from a Website.

The Setup

Perl scripts are simply plain old run-of-the-mill ASCII text files. However, instead of containing sentences that make sense to humans, they mostly contain commands that make sense to servers. There is an advantage to this though. Because Perl scripts are ASCII text files, the Perl programmer can also put human-understandable instructions in the script, and many often do, locating this text right at the beginning (top) of the file. It is easy to spot the information that is meant for you to read because the line will start with # character.

The # character is the "comment" character in Perl. Any line that begins with a # does not get executed by the server. (There's one exception to this which you will see shortly.) In addition, the programmer can also put a # character after a script statement to add comments. For example, you could see the following statement in a script which acts as a user-settable option:

$counthits = 1;           # 1 = Yes     0 = No

Once you have downloaded and un-ZIPped a script that you would like to try, you should use a text editor like NOTEPAD to open the main, or if found the configuration, .PL file and check the top of the file for any information or setup instructions. If there is a readme file, open that in a text editor and look for setup instructions also.

One key piece of information you should find either in the comments in the script file, or in the readme file, is the HTML tag you need to add to your Web pages to execute the script. Using the file transfer example in the next section, the tag for Bestdam Logger Lite would be:

<!--#exec cgi="/cgi-bin/bdlogger/bdlogger.pl" -->

Note that the provided HTML tag with some scripts may assume you are putting the script in the cgi-bin sub-directory, not a separate sub-directory under it. If this is the case, and you want to put the script in it's own sub-directory just modify the tag. For example, say you downloaded a script called "GigCount" and the tag specified in the script's comments or documentation was:

<!--#exec cgi="/cgi-bin/gigcount.pl" -->

If you wanted to put the script in it's own sub-directory named "counter" you would simply modify the tag to

<!--#exec cgi="/cgi-bin/counter/gigcount.pl" -->

CGI scripts that do not require SSI might have a more common type of tag. For instance, a script to take on-line polls may use a link to execute the script. In this case the tag would be something like

<a href="/cgi-bin/voter/castvote.cgi">Vote here</a>


In addition to a tag, the information near the top of the script file or readme file should also contain instructions for you on what values to use to CHMOD the files (i.e. set the permissions), as well as setting any user-settable options the script may offer.

The very first line of any Perl script is a user-settable option and is always going to be the path to your host's Perl installation, preceded by the characters "#!". This line is commonly referred to as the "shebang". Typical shebangs can be:

Note that Perl 4 scripts will work with a path to a Perl 5 installation but the reverse may not be true. If you're having problems getting a script to work with the first one, try the second. Note also that this shebang line may not be necessary with Windows servers.
If you haven't yet done so, now would be a good time to snoop around your host's or ISP's Website Technical Support pages and FAQs looking for anything related to "CGI". There you may find not only the path to their Perl installation (and possibly the version of Perl they have installed), but to their e-mail programs and other paths as well. If you can't verify this information, just leave it at the default value, but verify it with your host or ISP if you run into problems trying to use the script.

If you know how to use the telnet program to access the shell of your UNIX/Linux server, you can use the   whereis   command to find out the paths to your Perl and sendmail installations. At the shell prompt, simply type in
whereis perl
and
whereis sendmail
and the paths will be displayed. (If whereis gives you an error message try using which in its' place.) Note that you will often get multiple paths displayed, some ending with things like   /perl5.003   and   /sendmail.cf   but you are only interested in the paths that end with   /perl   and   /sendmail   - i.e. with no extensions. There may even been multiple path listings to these, but that just means there are different versions installed.

You may also be able to find the system path (see the paragraph below) to your root Web directory by using the   pwd   command (print working directory).
Setting options usually involves entering values for script variables. These values can be a '1', 'Y', 'y', 'YES', etc. to enable an option and a '0', 'N', 'n', 'NO', etc. to disable it. Certain user or system information may be needed for some variables. You may be asked to enter path information or an e-mail address. For example, near the top of the Bestdam Logger Lite file you are asked for your e-mail address (so the site visitor data can be e-mailed to you) and the path to your server's e-mail program. The comments in the script or configuration file should clearly indicate what the option is and what the valid optional values are.

Some scripts will ask you to enter the system path. This is not the URL. The system path is the path from the root of the server which is hosting your site and will look something like this:

/usr/local/etc/usersites/(your website identifier)/

You'll have to ask your host or ISP if you don't know what your path is. The problem is that they may not be too quick to give this information up for security reasons. If that's the case, there's not much you can do. You can try asking them to take a look at the script you want to use so they can see how the path is used.

If some of what you read at the beginning of the .PL file doesn't make any sense to you, don't feel like you're doing something wrong. Many Perl programmers will write these comments for other Perl programmers, or worse, for others who are well-versed in UNIX. If you find that the instructions are not clearly written for those without Perl programming or UNIX experience, and there is no accompanying readme file with easy-to-understand instructions, you may want to forget about using that script and find something with clearly written, understandable instructions.

Detailed setup instructions for Bestdam Logger Lite are given on the Setup & Installation page as well as in the readme.txt file contained in the .ZIP download file available on the Features & Download page. However, the top of the bdlogger.pl file is heavily commented with instructions for setting the options so you may be able to set them just by opening the file in a text editor and reading the comments.

With everything set up in the script file it's time to transfer the files and, with UNIX/Linux servers, set the permissions (detailed in the next section). The most common mistake people make when using ftp to transfer script files to the server is not using ASCII mode to transfer the files. Be sure to use ASCII mode when transferring the script files ! A script will not work if it is transferred in binary mode. You will see how to do this in the next section.

Troubleshooting

The vast majority of script problems are due to configuration issues. When you consider that there are so many different flavors of UNIX out there, multiplied by the number of possible configuration options, multiplied by the number of Web server software packages out there, multiplied by the configurations options they have, you can see why this is the case. That's not to say you should go running to your host or ISP if your script doesn't work right off. The truth is, trying to solve problems is one of the best learning experiences there is. There are a lot of things you can check, and if you do go to your host or ISP and say "I have checked....." they'll know you have some knowledge of what you're doing and take you seriously.

If you've set up and installed and script and you feel it may not be working, there are certain steps you can take using your browser to try and track down the problem.


Symptoms and possible causes of problems are outlined on the Bestdam Logger Help page but most of the information would apply to other scripts as well.

You may also have another troubleshooting tool at your disposal. If your host or ISP generates an individual error log for your domain, it will likely contain error messages which will indicate what problems were encountered when the server tried to execute the script. The error logs are typically just ASCII text files so you can ftp them to your local hard-drive and open them using a text editor.


Top of page



Transferring Files
Transferring script files to your Web server is not much different than transferring your Web page files there. The main differences are that you'll be putting them in their own sub-directories and, if your server is running a UNIX or Linux operating system, setting the permissions to the files.

The screen-shots in the steps below are using the WS_FTP program but other ftp programs should have similar functions.

  1. Go into the cgi-bin sub-directory (i.e. open the cgi-bin folder by double-clicking on it)


  2. Make a new sub-directory (folder) under cgi-bin with the name of bdlogger by clicking on the MkDir button.



    Technically, you could just transfer the files into the cgi-bin sub-directory but, because you may want to use additional scripts in the future, it is best to put each script in its' own sub-directory.


  3. Double-click on the new bdlogger folder to open it and   USE ASCII MODE !!!   to transfer all 4 of the following files into the new bdlogger sub-directory (the readme.txt file that comes with Bestdam Logger Lite does not have to be transferred):

    • bdlogger.pl
    • period.log
    • trigger.dat
    • pagehits.cnt




    NOTE:   If you are using any version of Windows, the file names may have been changed so that the first letter of the file names is upper-case. Make sure the file names are all lower-case letters once they are on the server. If they are not, use the "Rename" button of your FTP program to change the file names to all lower-case letters on the server.






    Setting File Permissions

    Web servers will generally be based on one of two operating systems. Either some "flavor" of UNIX or Linux, or Windows.  Step 4 is necessary only if your pages are hosted on a UNIX/Linux server.  Most are.  Check with your ISP if you're not sure of your server's OS.

  4. Once the files have been transferred you have to set file "permissions" so people can access them. The UNIX term for setting file permissions is CHMOD. You must CHMOD the Bestdam Logger Lite files as follows.
    bdlogger.pl - 755
    period.log - 666
    trigger.dat - 666
    pagehits.cnt - 666

    The above numbers are based on the "binary" number system used in computers. If you don't know what that means don't worry. You can CHMOD files using WS_FTP as follows:


    • click once on the bdlogger.pl file name to highlight it




    • right-click on the bdlogger.pl file name and select "CHMOD" from the pop-up menu to bring up the Remote File Permissions window




    • set the bdlogger.pl permissions to 755 by selecting the following in the three columns:

      • Under Owner - select Read, Write, and Execute
      • Under Group - select Read and Execute
      • Under Other - select Read and Execute





    • click once on the pagehits.cnt file name to highlight it

    • right-click on the pagehits.cnt file name and select "CHMOD" from the pop-up menu




    • set the permissions to the pagehits.cnt file to 666 by selecting the following in the three columns:

      • Under Owner - select Read and Write
      • Under Group - select Read and Write
      • Under Other - select Read and Write




    • repeat the previous three bulleted items for the remaining files

    • click on the Refresh button to the right of the server file display

    • click on the DirInfo button to the right of the server file display and you should have the following:

      • rwxr-xr-x     bdlogger.pl
      • rw-rw-rw-     pagehits.cnt
      • rw-rw-rw-     period.log
      • rw-rw-rw-     trigger.dat

      Note that the permissions for the file ending in .PL are set differently than the others. That's because the .PL file is the actual script file. The file that get's called by the tag in the Web page and executed by the Web server. That's why the Execute permission is needed for that file. The remaining files are log and count files that need to be updated. They are not executed, but merely written to with updated information. That's why the Write permission is needed for them.

      For future reference:   Let's say that you've been using a script and you find out there's a new version of the script file available. You obtain a copy of the new script file, set any required options, and you're ready to transfer it to the server. If you overwrite an existing file by re-uploading it you don't have to set the permissions again. The permissions are tied to the file name, not the file itself. So if the new version of the script file has the same name as your existing version, it's better to just re-upload it without deleting the old version first.

    • IMPORTANT:   If you created the cgi-bin sub-directory yourself because this is allowed by your host or ISP, use the above procedure to set the permissions to the sub-directory to 711 by selecting the following in the three columns:

      • Under Owner - select Read, Write, and Execute
      • Under Group - select Execute
      • Under Other - select Execute

      so that the Dir Info for the sub-directory is "drwx--x--x".


    Don't Assume !!!
    Many people assume that when they right-click on a file to bring up the Remote File Permissions window that what is displayed is the file's current permissions. This is NOT the case. What is shown is whatever settings were used the last time a file was CHMODed. You must use the DirInfo button to see what the current file permissions are.



  5. Now that you've got the script files ready to go, it's time to take care of the script's HTML tag. This tag may be an "SSI directive". Which pages need the tag depends on the script. Typically, most scripts only require the tag to be placed on one page.

    Important Note: Be careful about putting the script's tag on multiple pages. A script will usually only have a single data file for a given function. For example, if you install a very simple hit counter script and you put the tag on multiple pages, the "hit count" displayed could be the cumulative hits of all the pages having the HTML tag. You have to look at the documentation that came with the script. Some hit counters do have "multi-page support" for simultaneously tracking hit counts of multiple pages in one data file.

    Open your Web pages in your HTML editor and add the script's tag keeping the following in mind:

    • Make sure you put it before the closing "body" tag (</body>).

    • If the script is a hit counter or something else that displays something on your page, the placement of the tag in your HTML document will determine the placement of whatever's being displayed (hit count, etc.) on your Web page.

    • As mentioned in the previous section, your host or ISP may have the following requirements:

      • If you're using a script that requires SSI, the Web page with the SSI directive tag must have an .shtml or .shtm extension. Check with them regarding any such requirement. (This could present a problem if your page is already indexed or referenced on a lot of search engines or directories with a different extension.)

      • You may need to use a relative path in your tag. For example,

        instead of
        <!--#exec cgi="/cgi-bin/bdlogger/bdlogger.pl" -->
        use
        <!--#exec cgi="cgi-bin/bdlogger/bdlogger.pl" -->

      • Scripts may be required to have a .cgi rather than a .pl extension. This should not be a concern. It is normally not a problem to just rename the file (and make the corresponding modification to the tag) to comply.


    Once the tag is added to the appropriate page(s) and the changes have been saved, transfer them to your server as you normally would. i.e. Do not transfer them into the cgi-bin sub-directory or any sub-directory under it.

    Congratulations! You've just setup and installed a CGI script. That's all there is to it.

    If you don't want to try the script on your live Web pages right off, put together a test HTML page to try it out. Here's a simple test page to try out the Bestdam Logger script. You can copy/paste this HTML code into your HTML editor and save the file as cgitest.shtml, upload it into the same directory as the rest of your Web pages, and view it with your browser.

    <html><head><title>CGI Test Page</title></head><body>
    The hit count should appear bolded below this line<br>
    <b><!--#exec cgi="/cgi-bin/bdlogger/bdlogger.pl" --></b><br>
    </body></html>
    If you see a hit count displayed on the page you can use your ftp program to look at the script's data files (pagehits.cnt, period.log) to verify the script's proper operation. With WS_FTP, select ASCII mode, click once on one of the data files in the "remote" window, and then click on the "View" button. If you don't see a hit count, don't get all bummed out. As mentioned in the previous section, there are a lot of reasons a script may not work the first time. The Bestdam Logger Help page should help your resolve most problems.

    There are many free CGI scripts available on the Web that you can download and use. However, the reason I got into Perl programming was because I couldn't find a script that did exactly what I wanted done. Writing Perl scripts isn't that hard, and learning to write them means you can write scripts that will do anything you want them to do. The next section will show you the basics of the Perl language and some examples of what you can do with it.


Top of page



Messin' Around With Perl
If you've had even basic exposure to computer programming or Javascript you'll find that Perl is a very easy language to learn because of it's script nature. In addition, because Perl scripts are ASCII text files, you can look at the code of the scripts you use to see how things are done. This text-based script nature means that Perl is an "interpreted" language. i.e. compilation of a Perl script is done on the server every time the script is called. CGI scripts written in some other languages, such as C, have to be compiled manually (once) before they can be called. Even so, even large Perl scripts tend to compile/execute very quickly.

Don't let this ease-of-learning fool you. Perl is used for some very sophisticated applications. In addition, it can be used with the free MySQL engine for applications requiring the use of databases. While no one can predict the future, I would suspect that as Linux becomes more and more popular as an operating system for Internet servers, Perl will become more and more popular. If you invested the very short amount of time necessary to familiarize yourself with Perl, you could begin writing your own scripts (not to mention develop a highly marketable skill). There's even a Perl exam in several of the CIW certification tracks (Certified Internet Webmaster).

"Messin' Around With Perl" means just that. You don't have to start out writing your own scripts from scratch. Read the License Agreement or "conditions of use" that normally come with a script. Some script authors will say it is OK to modify their script. Because they're just text files you can take such a script, try using it for awhile so you know what it does, and then try customizing it a bit. Transfer your customized script to your Web server and see how your modifications affect the script's performance. (Keep a backup copy of the original script with your options set in case you hose things up. If you do hose it up, simply upload this backup copy to your server to replace the hosed-up version). The more you learn about Perl the more customizing you can do. That will help prepare you for writing your own scripts from scratch.

Caution:  Make sure you have a copy of Perl-Win32 on your local system (see the "Good Books" section below) so you can test your modified scripts. You don't want to try executing a script on a commercial server without doing some preliminary testing first. A rouge script could crash a server and since most hosting companies host a lot of domains on one server, bringing it down could make you very unpopular with a lot of Webmasters.

Perl Basics

If you know just a few simple Perl things you'll be able to look at most Perl scripts and figure out what they do.


Check it out for yourself. Here is the Perl script for a simple hit counter which uses the data file counter.dat to store the count value as ASCII text: 

#!/usr/local/bin/perl
print "Content-type: text/html\n\n";      # Web page as default output

  open (COUNT, "counter.dat");     # Open for read using COUNT file handle
  $hitcount = <COUNT>;             # Read in file to get current count value
  close COUNT;                     # Close the data file

  $hitcount = $hitcount + 1;       # Increment count value

  open (COUNT, "> counter.dat");   # Open data file for write
  print COUNT "$hitcount";         # Write updated count value
  close COUNT;                     # Close the data file

  print "$hitcount";               # Display updated count on the page

exit;
Notice that the only difference between writing something to a file and displaying it on a Web page is using the file's "handle" in the print statement. One thing you probably didn't notice is that there was no file locking done in this script. Bad news. Any time a file is opened for a write operation it should be "locked" so someone else can't try to write to it at the same time. The script should have a lock statement after the open-for-write statement (one of the things you want to look for in free scripts):
open (COUNT, '> counter.dat');
flock(COUNT, 2);
print COUNT "$hitcount";
Perl is very flexible and "programmer friendly". You don't have to declare variables or follow strict rules as to their "type". For instance, if a string variable contains a numeric digit you can mathematically add it to another string variable that also contains a numeric digit. Perl handles the type conversions automatically. One thing that does seem to give a Perl interpreter problems is the dash (-). It thinks you're trying to subtract things when it sees it so don't use dashes in names of variables, subroutines, etc. For example, you'd want to use $hit_count rather than $hit-count for a variable name.

Real World Perl

Perl started out as a reporting language so it has many features for handling text and strings. That includes text and strings that are components of an HTML document. As a result, you can use Perl scripts to dynamically create Web pages "on the fly". Consider this example:

Conditional Web Pages

Your corporate home page has a Human Resources link. You would like your company employees to see a different HR menu page from those who visit your Website from "the outside". You could use a script to read the visitor's IP address from the Apache REMOTE_ADDR environment variable and branch to the proper print statements accordingly. These print statements would then dynamically create the appropriate HR menu page. (Pages from search engines listing search results are often created dynamically and never written to an HTML file.) Assuming that your corporate sub-net is 199.299.x.x the main pieces of Perl code for this script, lets call it hrdirect.cgi, would be:


#!/usr/local/bin/perl
print "Content-type: text/html\n\n";
$ipaddr = $ENV{'REMOTE_ADDR'};
.
.
print "<html><head><title>MyCorp HR</title></head><body>\n";
print "<center><b>MyCorp Human Resources</b></center>\n";
print "<hr><p>\n";
if ( substr($ipaddr,0,8) eq '199.299.' )
 {
  print qq[<a href="http://hq.mycorp.com/news.htm">HR News</a><br>\n];
  print qq[<a href="http://hq.mycorp.com/forms.htm">Forms</a><br>\n];
  print qq[<a href="http://hq.mycorp.com/ben.htm">Benefits</a><br>\n];
  .
  .
 }
 else
 {
  print qq[<a href="http://www.mycorp.com/about.htm">About Us</a><br>\n];
  print qq[<a href="http://www.mycorp.com/jobs.htm">Jobs</a><br>\n];
  print qq[<a href="http://www.mycorp.com/app.htm">Apply Now</a><br>\n];
  .
  .
 }
print "</body></html>\n";
exit;


The    print "Content-type: text/html\n\n";    merely sets a Web page as the default output "device". Note also that the two "\n\n" (two line feeds) in the statement are both required. The blank line created by the second line feed acts as a data delimiter to Apache.

The  qq[...]  used in the some of print statements is just an optional way of quoting a string when that string itself contains quote marks. The output of the first group (internal visitors) of print statements would appear as follows in the HTML page source:
<a href="http://hq.mycorp.com/news.htm">HR News</a><br>
<a href="http://hq.mycorp.com/forms.htm">Forms</a><br>
<a href="http://hp.mycorp.com/ben.htm">Benefits</a><br>
Keep in mind that there is no HTML file associated with this Web page. It is created dynamically and only exists in the browser. If you wanted to create an actual HTML file with this information all you would have to do is add the appropriate file open and close statements to the script and add the file's handle to the print statements.

Note that "dynamically-created Web pages" is not the same as DHTML (DynamicHTML). DHTML pages can be updated while they are being viewed in the browser.
You could execute the above script by tying its' execution to the Human Resources link on home page. For example, a simple text link could be:

<a href="/cgi-bin/hrdirect.cgi">Human Resources</a>

When someone clicks on this link the script is called and it starts executing the print statements. The output of these print statements are HTML code which gets displayed as a new Web page in the browser. The page internal visitors would see would be:


MyCorp Human Resources

HR News
Forms
Benefits
 etc.


while external visitors would see:


MyCorp Human Resources

About Us
Jobs
Apply Now
 etc.


Note:  You wouldn't want to have links to sensitive information on the "internal" menu page. Server environmental variables can be faked. But for general employee information that would be useless to non-employees this could be a viable way of disseminating information.

Granted this is a pretty simple example but hopefully it's enough to give you some sense of what can be done. You could write a script to test for things like time of day, day of week, client platform, or any other host of variables and branch to different print statements to vary the content of the resulting Web page accordingly. You could also use additional print statements in your Perl script to add HTML code for headers, footers, company logo, e-mail links, etc. to the Web page.


On-Line Forms

Another common real-world use of CGI scripts is to process data thats collected using HTML forms. These form pages have elements (text boxes, radio buttons, check boxes, etc.) to collect visitor input. When a Website visitor enters information on a form and clicks on the "Submit" button the data is sent to the person collecting the information or stored in files.

A Web-page form has two parts:

  • An HTML page with form elements and buttons
  • A CGI script which is called by the HTML page's Submit button to process the form data
A "Contact Us" page with a form is very common so we'll use that as an example. One reason they're so popular is that you can have a drop-down list of subjects that the contact is in reference to (Product inquiry, technical support, Website problems, etc.) and the correspondence can be routed to the appropriate person depending on which subject was selected. Here's a simple example of such a form and the HTML code for it:


Your Name:

Your E-mail:

Select a Subject:

Your message:


The HTML code (the first piece) for this form is: 
<form method="post" action="/cgi-bin/contact.cgi">
Your Name:<br>
<input type="text" name="A_name" maxlength="50" size="30">
<p>
Your E-mail:<br>
<input type="text" name="B_address" maxlength="50" size="30">
<p>
Select a Subject:
<select name="C_subject">
<option value="none">-------------------------
<option value="sales">Product Information
<option value="support">Technical Support
<option value="web">Website Problem
</select>
<p>
Your message:<br>
<textarea name="D_message" wrap="physical" rows="6" cols="50">
</textarea>
<p>
<input type="submit" value="Send Message">
<input type="reset" value="Clear Form">
</form>
The <form> tag in the above HTML code shows that the contact.cgi script is what gets called when the submit button is clicked.

Note that you have to give each form element a unique name. That's because the Web server creates a "buffer" and stores these names, and their values entered or selected by the user, in them.

That's how the visitor's data is passed from the HTML page to the CGI script. The CGI script for a given HTML form page is programmed to read in this buffer and pull out the form element names and their values.

Notice that I gave each form element a name which started with an upper-case letter followed by an underscore (_). There's a reason for this. When the Web server creates the buffer and puts the form element data into it, it does so in no particular order. As a result, when the CGI script reads in the variable values they can be in any order. You'd like to present or store the information in the same order as it appears on the HTML form page.

By using the upper-case letters followed by a dash, we can then tell the CGI script to sort the variables and these upper-case letters will control the sort (and presentation) order. If your form page has a lot of elements and you run out of upper-case letters, you can start using the lower-case alphabet as they will sort after upper-case. Again, the HTML form page is only the first of two pieces that are need to process forms.

Now for the second piece. The CGI script to process the above HTML form is as follows. Note that the items in blue refer to form element names from the HTML page, and items in red refer to visitor input. 

#!/usr/local/bin/perl

$mailprog = '/usr/sbin/sendmail';

#-----------------------------------------------
# This first piece of code between the dashed lines is a
# generic 'get the data' routine that can be used with
# ANY form page.  It is from Jaqueline Hamilton's book
# CGI Programming 101 (ISBN 0-9669426-0-4).  It reads in 
# the buffer contents, converts Hex characters to ASCII,
# and puts them into variable pairs.  Each pair consists
# of the form element name (the variable) and its' value.
# This pair storage structure is called an array or, more
# commonly in Perl-ese, a hash.

read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
@pairs = split(/&/, $buffer);
foreach $pair (@pairs)
{
  ($name, $value) = split(/=/, $pair);
  $value =~ tr/+/ /;
  $value =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C", hex($1))/eg;
  $FORM{$name} = $value;
}
#-----------------------------------------------

# These IF statements pick a company e-mail recipient
# based on the Subject that was selected on the HTML form.

if ($FORM{'C_subject'} eq 'none')
{
  $sendto = 'kendra@mydomain.com';
}
if ($FORM{'C_subject'} eq 'sales')
{
  $sendto = 'stu@mydomain.com';
}
if ($FORM{'C_subject'} eq 'support')
{
  $sendto = 'araceli@mydomain.com';
}
if ($FORM{'C_subject'} eq 'web')
{
  $sendto = 'keith@mydomain.com';
}

# With the appropriate recipient determined we can
# send the text of the message to them along with
# the name that was entered on the HTML form.
# The open statement pipes the print output to
# the e-mail server software (usually Sendmail).
# The first three print statements are for the
# e-mail header (recipient, sender, and subject).
# The foreach/print loop prints each of
# variable/value pairs in sorted (alphabetical)
# order in the body of the e-mail message.

  open (MAIL, "|$mailprog -t");
  print MAIL "To: $sendto\n";
  print MAIL "From: $FORM{'B_address'}\n"; 
  print MAIL "Subject: Website Submission\n\n";
  foreach $key (sort keys(%FORM))
  {
    print MAIL "$key  =  $FORM{$key}\n";
  }
  close (MAIL);

exit;
This is a very basic form/script example and has not been tested. Ideally you'd want to enhance the script to make sure that a subject was selected and that a name and e-mail address were entered. If something was missing you would then have the script produce an error Web page (i.e. print HTML statements as in the HR example above) telling the visitor to go back and correct the error and re-submit. It is presented to show you the pieces involved in an on-line form and how they are inter-related.

Note:  One advantage of using a form mailer like the above on your Web site instead of simply having e-mail links is that spammers will often scan Web sites looking for and recording any e-mail addresses they find in the HTML. As a result, Webmasters (who's e-mail addresses are typically given on Web pages) often end up getting a lot more junk mail than average users. These scanning utilities cannot scan scripts in the cgi-bin directory so e-mail addresses in those are not picked up.

Real World SSI

You've seen the SSI exec directive in action in the Bestdam Logger tags in previous sections but there are other SSI directives. A neat one is the include directive. It allows you to include the contents of a text file to become part of your Web page. Consider this example.

The people in your organization would like to view a report containing some real-time data. If the data is in, or you can convert it to, an ASCII text file you could easily create a Web page version of this report. People could view it from anywhere using a browser and no Perl script is needed. Simply add a tag with an SSI include directive to a Web page to "suck in" the external data file whenever the page is viewed. That's an important point because the actual reading of the text file is done when the page is requested. That's what makes it useful for real-time data reporting. Just create an HTML report page with any company logo, footer, e-mail links, etc that you want and add the SSI include directive in the area of the HTML where you want the text to appear.

There's one other thing that will make your job a lot easier and that's the <PRE> (preformatted text) HTML tag. Enclosing the SSI directive within a set of <PRE> tags will cause the text to appear on the page exactly the way it appears in the file. You don't have to use a lot of other HTML, such as &nbsp and <br> tags, to try and replicate the file's appearance under HTML. For example, if the real-time data is in a formatted text file called rt-data.txt, a simple Web page to display it would be

<body>
some html
<PRE>
<!--#include file="rt-data.txt" -->
</PRE>
some html
</body>
Likewise, you wouldn't want to use this for any sensitive company information or data which contains Social Security numbers, etc. But it may be useful in cutting down on some of the paper flows in your organization.

Note that each Web-server software package tends to have it's own set of SSI directives. The directives used here are for the widely used Apache Web-server software that runs on most Linux servers. Another thing to keep in mind is that if your preformatted text file containing the data also includes certain characters that HTML is sensitive to you may get a weird display. For example, don't enclose anything in the data file in angle brackets (< >). Your browser will still try to read it as a tag even though it's between the <PRE> tags.

A more common use of the include directive is to read in files containing HTML. On most Web sites, the top of the page is the same on every page. Or every page contains a side-bar menu. It would be a pain to have to edit every page when a change needs to be. It's easier to put this common HTML into a text file and just put tags containing the include directive where the HTML would normally go.

<body>
<!--#include file="header.txt" -->
page-specific html
<!--#include file="footer.txt" -->
</body>
These examples contain Perl, HTML, and SSI material so it may be a bit overwhelming if you've never worked with this stuff before but I wanted to use examples that could be useful to you at some point in the future should you decide to pursue Perl programming and SSI.

You may even discover more sophisticated uses where you could combine the above two capabilities. For example, use a script to create an HTML file and use script logic to add one of several different SSI include directive tags to the HTML file (the script output) to bring in a certain ASCII text file when the HTML file is viewed. What you would, in effect, be doing is setting up "conditional" SSI directives. Depending on what conditions the script finds, a corresponding SSI directive is added to the HTML file.

If you understand the above, congratulations, you understand Apache's new XSSI (eXtended Server Side Includes) where SSI directives are executed by the server only if certain conditions exist, without the need of a Perl script.
However, don't try to add an SSI directive tag to a dynamically-created Web page (i.e. one that's never written to a file) because the server doesn't "parse" (look through) script output. As a result, the directive would never be detected and processed. There may be a way to get an SSI directive in a dynamically-created page to work but I haven't been able to find it.

As you can see, Perl and SSI have many neat features too abundant to cover here. Suffice it to say that you can be writing Perl scripts within hours, not days, of reading up on it, and as your skills grow you'll be amazed at what you can do with it.

Another benefit of learning Perl is that some of what you learn also applies to UNIX/Linux commands and shell scripts which makes learning these OSs easier. The use of > vs >> and single quotes vs double quotes are just a couple examples. Shell scripts are like DOS batch files. They are text files with a series of shell commands. However, you can also write more sophisticated shell scripts using Perl.

Good Books

Of all the books I've bought on Perl (about 12 of them) the following three are the most valuable and useful. They are excellent as either tutorials or in their completeness in their coverage of the subject. Clicking on a book's cover will take you to Amazon's "detail" page where you can read an overview of the book and comments posted by those who have purchased it.

The comments next to the book covers are my own and they are my honest opinions, not a sales pitch.


Perl For Dummies
More info... 		The best book out there for beginners, including those who don't know a lot about programming. Perl for Dummies comes with a CD that includes sample code and a copy of Perl-Win32 that you can use on your Windows-based system to test your scripts. The book covers using the Perl-Win32 software also. It doesn't get into the real fancy Perl functions but that's not its' goal. It's an "easy read" that will have you coding in no time.
Mastering Perl
More info... 		If you're an experienced programmer this may be a good starting point for you. Mastering Perl is very comprehensive in it's coverage of Perl, addressing functions that the "Dummies" book doesn't get to. The information is presented in a very easy-to-understand manner and doesn't assume you have a UNIX background. (A lot of the more advanced books do.) Part VI of the book (four chapters) is dedicated to Perl/CGI on Windows servers. They also cover Blat and Sendmail for Windows.
Effective Perl Programming
More info... 		I wish every programming language had a book like this. Effective Perl Programming is a unique work that will make a Perl programmer a good Perl programmer. It's a compilation of "items" that one usually only finds out with many years of experience. He also covers not only using modules, but how to write them as well. It's not a ground-floor tutorial like the "Dummies" book, but if you're going to learn Perl, you might as well learn to do it right. This should definitely be the second Perl book you read.
Go to the Amazon.com Bookstore If you're going to Amazon to shop for something other than the above titles, please use this link to go there.


Creating a custom 404 page to match your site's appearance is easy!
See how on the About htaccess and XBitHack page.



Did you find this page helpful ?
If so, please use the Amazon book links to help pay
the costs associated with making this page available.


Top of page




Contents, diagrams, and images    Copyright © 2004-2023    Keith Parkansky    All rights reserved.
Certain graphics, symbols, and terms used on this site and in its documents are registered trademarks
of their respective owners and are contained herein for identification purposes only.
No endorsement of this site, its contents, or its documents by these owners is expressed or implied.

LIABILITY
IN NO EVENT WILL KEITH PARKANSKY BE LIABLE TO ANY PARTY (i) FOR ANY DIRECT, INDIRECT, SPECIAL, PUNITIVE OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, DAMAGES FOR LOSS OF BUSINESS PROFITS, BUSINESS INTERRUPTION, LOSS OF PROGRAMS OR INFORMATION, AND THE LIKE), OR ANY OTHER DAMAGES ARISING IN ANY WAY OUT OF THE AVAILABILITY, USE, RELIANCE ON, OR INABILITY TO USE THE INFORMATION, METHODS, HTML OR COMPUTER CODE, OR "KNOWLEDGE" PROVIDED ON OR THROUGH THIS WEBSITE OR ANY OF ITS' ASSOCIATED DOCUMENTS, DIAGRAMS, IMAGES, REPRODUCTIONS, COMPUTER EXECUTED CODE, OR ELECTRONICALLY STORED OR TRANSMITTED FILES OR GENERATED COMMUNICATIONS OR DATA EVEN IF KEITH PARKANSKY SHALL HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES, AND REGARDLESS OF THE FORM OF ACTION, WHETHER IN CONTRACT, TORT, OR OTHERWISE; OR (ii) FOR ANY CLAIM ATTRIBUTABLE TO ERRORS, OMISSIONS, OR OTHER INACCURACIES IN, OR DESTRUCTIVE PROPERTIES OF ANY INFORMATION, METHODS, HTML OR COMPUTER CODE, OR "KNOWLEDGE" PROVIDED ON OR THROUGH THIS WEBSITE OR ANY OF ITS' ASSOCIATED DOCUMENTS, DIAGRAMS, IMAGES, REPRODUCTIONS, COMPUTER EXECUTED CODE, OR ELECTRONICALLY STORED, TRANSMITTED, OR GENERATED FILES, COMMUNICATIONS, OR DATA. USE OF THIS SITE CONSTITUTES ACCEPTANCE OF ALL STATED TERMS AND CONDITIONS.