Our customer was a local town that had a custom collections system for tax and utility bills, and parking tickets that was built on top of an Informix database and was written in Informix 4GL. The Treasurer's Office was quite happy with its performance and wished to keep the system but wanted to add the ability to make bill payments online, via the Internet.
BSM Development created all of the Web-based software to display bills from the Treasurer's Web site in real time by extracting billing information from the Informix database and formatting it for display by a Web browser. The capability to look up real estate and excise tax bills, water bills and parking tickets was provided as well as the ability to look up historical bills (i.e. paid bills). Because the 4GL code could not be used to implement Web billing, new code had to be written (in Perl) and much of it had to be reverse engineered from the existing system, since the original code was written without documentation and the programmer in charge literally died on the job, taking all knowledge of the system with him to the grave.
Third party payment processing services were evaluated and a vendor was chosen. BSM Development worked with this vendor to produce a secure method of transmitting transactions to the vendor's Web service via SOAP. The SOAP portocol encodes all of the transaction information in XML form and the XML request is then transmitted via an encrypted tunnel over the Internet to the Web service, which responds in real time with an answer to the transaction request.
The town's Web site was set up to include a "Pay" button on each bill which collected and validated all payment details. The payment transaction was passed to the the vendor's Web service and the results returned in real time. An interesting wrinkle was that the town was the originator of a program that allowed taxpayers to include a voluntary scholarship fund payment on their tax bills so this needed to be incorporated into the bill payment page.
Two types of transactions were supported: ACH payment; and credit card payment. Real-time post back of payment transactions to the customer's account was possible but only during certain periods of the day. During actual business hours, it was not possible to post back payments due to the need to carry out the daily auditing of the database, required by law. Consequently, a queue was implemented to hold transactions entered during the auditing window in abeyance until the close of business, at which point they were posted. However, the account was updated with a pending payment so that the customer would see their payment immediately.
When it actually came time to post the payments, either directly during off hours or as a result of the queue of delayed payments being run after close of business, there were two routes that could be taken. Credit card transactions, being real time, were posted immediately. However, ACH transcactions required a two step process. The ACH transactions could only be completed early in the next day, after the payment processing service had actually transmitted the information to the banks involved and received the OK back from them. This necessitated that a nightly resolution job be created that could resolve outstanding transactions and roll back any failed ones.
Given that the nightly resolution job had to run early the next day anyway, the opportunity was taken to include in it all of the batch reporting that was required for transaction auditing purposes. A set of reports was generated and automatically emailed to all those concerned, listing all of the previous day's successful and failed transactions. We find that automatically emailing reports to responsible people is a good way to monitor the operation of otherwise unattended systems.
At the same time that the online bill payment system was implemented, we also built a system to electronically process batches of bill payments that were recieved from a third-party bill collection service. Much of the code that was written for the online bill payment system was reused to implement the second system in very short order.
One of the lessons learned is that extraordinary efforts must be made to guaranty the reliability of the payment process, especially when remote payment processing is involved. Transactions must either be guaranteed to work or guaranteed to fail. There can be no middle ground (e.g. the money came out of the user's bank account but never got posted to their tax bill). Provision must be made to clean up missing transactions, either through resolution at some point in the future or through two-phase commits. The ability to cleanly roll back failed transactions is a must.
Another lesson learned is that the system must be reliable and robust. The customers are depending on the system to make payments, often at the last minute. The system must be highly available and it must be reliable. All attempts to recover transactions instead of failing them should be made. To this end, we employed a resolution process that would retry failed transactions over several hours to compensate for temporary communications failures, system down time, etc.
A third lesson learned is that it is of paramount importance to have up-to-date documentation of your critical systems. Said documentation should be maintained on an as-you-go basis. As horrible as it is to contemplate, one can never tell when a key member of a system's team may become unavailable, taking their knowledge with them. To have to reverse engineer legacy code is not a pretty sight.
One bright spot was the ease with which the second system for processing payments from the third-party bill collection service was implemented. The modular, API-oriented approach that we used in building the payment handling routines allowed the second system to, basically, be constructed for the price of the first one. It can be said that the time spent in doing it right the first time was more than worth the results.