IWDN Email Form handler

[inimino]

Should IWDN create a form processor?

What features should it have?

Do we have any volunteers to contribute development effort, graphics, documentation, etc?

What form processors already exist and what's wrong with them?

I'm willing to spend some time on development if there's a consensus that we can meet a need that isn't already met by an existing free script. I haven't looked at the existing scripts that are out there though, so I'm curious to hear more from those who have, as well as any other thoughts you all may have.

[Odd Fact]

My programming skill are not the best but I will help where I can.

configurable multi send to emails
spam control
as many fields as needed
multi page maybe
good documentation (I can try and help here)

Some mailers are time consuming to work with due to lack of information.

[the_pm]

I'm down for the cause.

I can do any visual design components that need to be incorporated. While I don't imagine there would be many (if any), I have experience with HTML emails, what markup works in which mail clients and whatnot. I would classify this under design.

Feature List:
- Injection protection (practice safe mail() )
- Email address obfuscation - no sensitive information should be required within the HTML form that is used to send info to the processor.
- Q&A challenge/response anti-spam mechanism, must accept multiple correct responses.
- CAPTCHA (I don't like CAPTCHA, but many people would find it valuable to be able to choose between the two anti-spam mechanisms).
- Option to email a copy to the sender.
- Option to include custom text along with the copy sent to sender.
- Email multiple recipients
- Send attachments
- Custom error messages (required fields not filled in)
- Variables that allow you to set paths to header/footer include files. Make it somewhat skinnable, for the sake of error messages and...
- Success message - option to place in script as a variable (echo " HTML "; ) or redirect to a success page.

That's a quick list in no particular order. I'm sure there's more, but this is a decent start, I think.

Existing Scripts:
- FormM@iler - dbmasters.net/index.php?id=4 - Functionally, this script is pretty darned good. The biggest concern here was sloppy markup, IIRC. It's missing verification techniques for form spam control.
- FormMail and its clones - scriptarchive.com/formmail.html - I don't know that there's anything useful to say about this, but it exists.
- CGIEmail - sfsu.edu/training/cgiemail.htm - Ok, I used to LOVE this application! Holy crap, it was nice from the user standpoint! It's also about as insecure as a script can get. You simply created an email template using [ItemName] as objects in the email that were replaced by stuff submitted in the form. True, anyone could exploit it. True, the script sent tons of spam all over the Web, but minus this part, it was a Godsend of a script!

[crazyfish]

This is a great idea.

Attachments and anti-spam would be great. I use the FormM@iler and those are missing from it.

[the_pm]

This is a great idea.

Attachments and anti-spam would be great. I use the FormM@iler and those are missing from it.(psst. FormM@iler does support attachments, even multiple - it's one of the only ones ;) )

[crazyfish]

(psst. FormM@iler does support attachments, even multiple - it's one of the only ones ;) )

Really??? *Off to read the features...*

[the_pm]

I think one of the big areas where you're going to see problems isn't so much in the functionality, but moreso in the quality of the markup. I'm going to assume most form handlers have evolved over the course of years and have grown increasingly sloppy. Having a pristinely coded handler makes it easier for coders to alter and bolt on new functionality later.

Also, all of the components that a script like this needs are available out there. I've not seen a free script that had them all nicely packaged together into one application, particularly with good anti-spam measures.

CAPTCHA and Q&A are just two possible anti-spam mechanisms. Others might include:

1. IP-based flood trap - If an IP sends X# of messages through a form over X period of time, the IP is banned from the site (requires writable .htaccess) and the site administrator is sent an email.

2. Secondary verification page - Having a review/send intermediary page will stop many bots dead in their tracks.

3. Form element tests - a small bundle of common form element tests will discourage spammers. For instance, the bundle might include a test that ensures email addresses contain @ and at least one '.' with no more than four characters trailing it. A phone number test would fail if the field had less than 10 characters (or a suitable minimum that can be applied worldwide). Postal code would have a minimum. There are others, I'm sure.

4. Keyword Filter - the user could input a number of keywords into a filter that would cause automatic failure if any field contained any of those words. Even better, the filter could be applied on a per-element basis.

5. Failed Attempts IP block - A script could block an IP after X# of failed attempts. But would this require a database to track?

I'm not saying this script needs to have all of these, or heck, even any of them (no need to include unnecessary bloat). I'm just brainstorming :)

[Odd Fact]

I agree Paul, the sloppy code makes modifying to your needs difficult. Maybe a plugin system like wordpress for the mailer?

[inimino]

Some mailers are time consuming to work with due to lack of information.

I agree. In my experience documentation is a weak point of many smaller software projects. Good documentation would be a big plus. If the software is simple to use and the documentation can be short, all the better, as long as it's well-written, comprehensive, and easy to find. I think we can pull that off with a little effort.

I'm down for the cause.

I can do any visual design components that need to be incorporated. While I don't imagine there would be many (if any), I have experience with HTML emails, what markup works in which mail clients and whatnot. I would classify this under design.

That would be great. It would be great to have a nice-looking home page for the software too.

[the_pm]

I don't know that a plug-in system is necessary (maybe it is?). My only point is that having a handler with clean markup would be worthwhile all by itself. There's of coure a lot more to consider, but even if that was all that would come of it, it would be a Good Thing (TM)

[inimino]

1. IP-based flood trap

As a stop-gap of last resort, probably worthwhile.

3. Form element tests

Yes. We could borrow a little from XForms and Web Forms 2.0 here.

4. Keyword Filter

Maybe better for the user would be to have an appropriate spam filter on their email inbox.


- Q&A challenge/response anti-spam mechanism, must accept multiple correct responses.
- CAPTCHA (I don't like CAPTCHA, but many people would find it valuable to be able to choose between the two anti-spam mechanisms).


Security is an issue with these scripts, because they are so widely deployed and of such varying quality. Perhaps we can avoid exploitable holes with some care and a few sets of sharp eyes on our code. (I've also seen one or two PHP projects that seem to suffer from the misconception that security is something you can apply in successive layers, like paint. That's not how it works.)

Abuse of the form is a related issue, but not exactly the same. If someone sends unwanted email using the form, arguably the form is doing what it was meant to do. I wonder if this is primarily what the CAPTCHAs and challenge-response systems are trying to prevent.

I dislike CAPTCHAs and similar approaches. They seem to me like an ugly hack to avoid solving some real underlying problem, which invariably persists. Of course, solving underlying problems is usually more difficult than applying an apparent quick fix. I'd be curious to know what these would be used for, if anyone thinks we should include this feature.

The fatal problem with CAPTCHAs for me is that they are invariably inaccessible to users with disabilities (yes, even the ones with those little audio links). That's a total non-starter for me, though it's just one large symptom of a fundamentally wrong approach to a problem.

[inimino]

I don't know that a plug-in system is necessary (maybe it is?). My only point is that having a handler with clean markup would be worthwhile all by itself. There's of coure a lot more to consider, but even if that was all that would come of it, it would be a Good Thing (TM)

A plugin system like WordPress might be a bit heavy.

Another interesting idea would be to have a builder on the site that includes the features the user chooses and no others. A script that has only the desired features would be easier to understand and modify if necessary.

[inimino]

This is a great idea.

Attachments and anti-spam would be great. I use the FormM@iler and those are missing from it.

Hm... attachments will add some complexity. It might not be unreasonable though, if there's demand for that feature (and I can see where there might be). Who needs/uses this?

[the_pm]

Hm... attachments will add some complexity. It might not be unreasonable though, if there's demand for that feature (and I can see where there might be). Who needs/uses this?Attachments was the entire reason I turned to FormM@iler in the first place. I needed a handler for a site that used a form to submit employment application information, and the form needed to allow a word document and a picture to be included. Anything that didn't handle attachments was instantly off my list. Perhaps digging into FormM@iler's markup will reveal a good path to follow? IIRC, the only thing that's needed within the actual HTML form to activate attachments (apart from declaring the form element names within the script) was to set a proper enctype.

Maybe better for the user would be to have an appropriate spam filter on their email inbox.Maybe, but I can see where an optional keyword filter at the sending point would be useful, 1) for creating a good IP trapping algorithm, and 2) in the event the email recipients aren't prepared to filter whatever the form might send them. If you cut off the problem at the source, then one set of filters will prevent the server from ever having to allow the message into its POP mechanism, and will prevent multiple people from having to set up filters. One form could service one person or potentially hundreds of people. The ability to turn on a keyword filter would be a pretty nice feature. I'd be game for personally creating a default filtering list for anyone who wants to have this working out of the box.

[the_pm]

BTW, when I've mentioned spam earlier, I was really referring to form abuse. I'm not too worried about us having the knowledge to prevent injection.

[Tyler]

...CAPTCHA and Q&A are just two possible anti-spam mechanisms. Others might include...

One of the most effective, simplest, non-intrusive methods I have found is simply require a "department" to send it to, making the default checked unacceptable (ex: "choose one..."). I haven't got any spam, and I use it on a few websites.

May not be the best solution, but I'm all for simplicity and non-annoying methods. :thumbsup:

[the_pm]

One of the most effective, simplest, non-intrusive methods I have found is simply require a "department" to send it to, making the default checked unacceptable (ex: "choose one..."). I haven't got any spam, and I use it on a few websites.

May not be the best solution, but I'm all for simplicity and non-annoying methods. :thumbsup:It falls short, unfortunately. I've been using the exact method you described for a while, and it stops a lot of spam, but not all of it. Some bots will randomly select a value for selectable items. On our own site, we still get 2-5 form spam messages every day, down from roughly 50. It's just too easy to defeat.

IWDN used to get 50-100/day, sometimes more. The Q&A challenge question ended that completely :)

[Corey Bryant]

A bit late (as usual) but I fully agree. I just did something for the Expression Blog for the JMail email component and .NET is my next step.

I did refer people of course over here for the PHP help but I fully agree. With the changes being made at Microsoft and their new editor (consideration of (more) support in PHP, etc) in the next versions, this would probably be a great time

[inimino]

Attachments was the entire reason I turned to FormM@iler in the first place. I needed a handler for a site that used a form to submit employment application information, and the form needed to allow a word document and a picture to be included.

Hm, ok. Are you still using that feature? Are others here using it? Perhaps it's necessary, I simply have no idea how common that kind of usage is.

Perhaps digging into FormM@iler's markup will reveal a good path to follow?

The issue isn't the markup, it's the code on the server to handle the attachments. FormMailer simply attaches them to the email and fully trusts the recipient to handle whatever unknown content was submitted. Whether that's a feature or a gaping security hole is perhaps a matter of perspective and of how the form is used.

IIRC, the only thing that's needed within the actual HTML form to activate attachments (apart from declaring the form element names within the script) was to set a proper enctype.

Yes, but then the server gets hit with some number of files of unknown size, origin, and content. Plain text email or email using a subset of HTML is much safer than mail containing arbitrary files uploaded by untrusted users. Attachments violate many of the assumptions people make about mail from a form, which is often whitelisted or otherwise processed differently from normal mail.

If the feature is necessary, I suppose I have no problem with it as long as it is optional and we note the security considerations.

Maybe, but I can see where an optional keyword filter at the sending point would be useful, 1) for creating a good IP trapping algorithm, and 2) in the event the email recipients aren't prepared to filter whatever the form might send them. If you cut off the problem at the source, then one set of filters will prevent the server from ever having to allow the message into its POP mechanism, and will prevent multiple people from having to set up filters. One form could service one person or potentially hundreds of people. The ability to turn on a keyword filter would be a pretty nice feature. I'd be game for personally creating a default filtering list for anyone who wants to have this working out of the box.

Would it be worth integrating some existing spam filtering technology, as an optional feature or extention? Spam filtering has moved on quite a ways by now from naïve keyword filters, and my expectation would be that if we implemented a keyword filter and it was actually used, it would attract new feature requests until we have reimplemented SpamAssassin. (Although I think a more likely outcome is that it just wouldn't be widely used.) However, if we integrated some existing effective filtering software at the source, it might be a worthwhile feature for high-volume forms, forms that attract a lot of spam, or forms that send email to many accounts.

[the_pm]

Hm, ok. Are you still using that feature? Are others here using it? Perhaps it's necessary, I simply have no idea how common that kind of usage is.Yes, I still use it. I believe the handler allows you to specify extentions allowed.

Whether that's a feature or a gaping security hole is perhaps a matter of perspective and of how the form is used.Considering it can be turned on or off by the Webmaster, and it's off by default, I'd say it's a feature. I can see plenty of possible uses for an email handler where the users have already been authenticated in some manner, or where the Webmaster uses antivirus software on the server.

Would it be worth integrating some existing spam filtering technology, as an optional feature or extention? Spam filtering has moved on quite a ways by now from naïve keyword filters, and my expectation would be that if we implemented a keyword filter and it was actually used, it would attract new feature requests until we have reimplemented SpamAssassin. (Although I think a more likely outcome is that it just wouldn't be widely used.) However, if we integrated some existing effective filtering software at the source, it might be a worthwhile feature for high-volume forms, forms that attract a lot of spam, or forms that send email to many accounts.That's definitely a possibility. I don't know enough about integrating third party scripts to know how easily or effectively this would work, but it sounds very interesting :)

[inimino]

Considering it can be turned on or off by the Webmaster, and it's off by default, I'd say it's a feature.

It seems like this is worth having. If it can be optional (so that the code to do this is not even included or installed if it is not wanted) I think that would be best.

That's definitely a possibility. I don't know enough about integrating third party scripts to know how easily or effectively this would work, but it sounds very interesting :)

If we do this I think it should also be optional. It would require some kind of storage on the server, either a database or flat files, and some kind of quarantine for submissions identified as spam, so it would add some amount of admin UI.

The form owner needs to be notified of quarantined messages, e.g. emails from successful form submissions could include a note "additionally there are 47 submissions in the spam quarantine". There would need to be an interface for viewing quarantined submissions and either deleting or authorizing them. Successful emails might also include a "classify this as spam" URL.

I intend to take a survey of the most popular form processors and see what kind of features they have.

[the_pm]

It seems like this is worth having. If it can be optional (so that the code to do this is not even included or installed if it is not wanted) I think that would be best.How about if the code is commented out and can only be activated by physically uncommenting every applicable line?

I was hoping to avoid databases with this one, because a whole new level of complexity is added when databases become necessary. Would flat files need to be encrypted, or would there be a strong enough authentication scheme to protect them from prying eyes? Which direction makes more sense (or maybe both)?

[inimino]

How about if the code is commented out and can only be activated by physically uncommenting every applicable line?

We might want a builder which includes or excludes optional features and then generates a pre-configured package for download and installation. I think it's best if non-programmers can use the script without having to touch the code.

I was hoping to avoid databases with this one, because a whole new level of complexity is added when databases become necessary. Would flat files need to be encrypted, or would there be a strong enough authentication scheme to protect them from prying eyes? Which direction makes more sense (or maybe both)?

Flat files are no less secure than a PHP file containing database credentials, so we wouldn't need encryption. Alternatively, PHP5 now includes an SQLite driver by default, so we can have a database without any user configuration. I'm not sure yet which makes more sense, and there might be other features that would benefit from some storage on the server which would affect our decision so we'll probably have to come back to it.

[the_pm]

You bring up a good point, MJ. PHP4, PHP5 or PHP 4/5 compatible?

Even though PHP4 is EOL, a LOT of hosts have not made the jump yet, or at least are not running both 4 and 5. Now, without knowing the specifics on how the script will work, I don't know how difficult it will be to cater to both. I just thought I'd throw that out there as a consideration.

Alternatively, PHP5 now includes an SQLite driver by default, so we can have a database without any user configuration.I'm still somewhat (though not wholly) resistant to going the database route, as this is something that's monitored and leveled at the root, and I would hate to see people having to upgrade hosting, make special arrangements with a host or even have to leave one host for another because of this (I migrated a hosting customer of mine away from paid Homestead ($200/yr.), because his new site required databases and Homestead doesn't allow them, not even for a fee!).

[Odd Fact]

The upgrade to PHP5 is why I started looking for a new mailer. The one I used was not compatible and fixing was too much work.

[inimino]

Now, without knowing the specifics on how the script will work, I don't know how difficult it will be to cater to both.

I don't know either, it will depend on what kind of features we want and how we might implement them.

PHP 5 is the reasonable target, and if PHP 4 support comes for free or with very little extra work, that would be great, but if it's going to constrain design decisions or feature set, I think that would be unwise. I don't know how many hosts are still running only the obsolete PHP version, but they can only become rarer now that the EOL is official.

I'm still somewhat (though not wholly) resistant to going the database route, as this is something that's monitored and leveled at the root, and I would hate to see people having to upgrade hosting, make special arrangements with a host or even have to leave one host for another because of this (I migrated a hosting customer of mine away from paid Homestead ($200/yr.), because his new site required databases and Homestead doesn't allow them, not even for a fee!).

Are you familiar with SQLite? It doesn't require any installation or attention from the server admin. All it requires is that PHP is compiled with support for it, which it is by default since 5.0. It is probably a better fit than MySQL for the majority of PHP projects that use a database.

However, I do think any feature that requires a database or any other server-side storage should probably be an optional feature. The simplest case of just a form and a static script that stores nothing on the server might be worth supporting.