Just realized one of our major mailing list signup forms on Harpoon has been busted for who knows how long. Explains the major drop in our conversion rate recently. Ugh.
Plus we just did an ad campaign with this thing busted. Double ugh.
And there’s really nothing about the form that makes it obvious it’s broken, so people probably think they’re signed up, but will never hear from us again. Triple ugh.
Excuse me while I go puke.
Do folks here with web apps use an automated testing tool such as Selenium to catch such problems? We use it at my day job to test new versions of our web app.
Great tip @steve. We’ve used automated testing tools on other projects, but didn’t think we needed to go through the hassle with some simple signup forms. Thinking differently about that now. 
If you have an active running form with consistent conversions, setup a Google Analytics alert that sends you an email when conversion rates drop under a certain percentage.
I use GA alerts for all sorts of site health checks. Traffic drops from consistent sources. Sudden surges indicating a possible media mention.
There’s a delay of a day or more, so they shouldn’t be your first indicator of a major problem, but it’s a cheap and easy backup solution if you already use Analytics.
Related to this we started building some functional tests into our system but one of the problems I had was things like the purchase flow on our production checkout process. How do people check production flows that touch things like inventory and purchasing without messing up all the data with constantly refunding the bot that keeps buying stuff!
Didn’t know you could set alerts like that in GA. Cool 
Yeah I make heavy use of selenium to test those multi step flows from trial to onboarded. The most difficult thing I found when setting it up is how to deal with the email notification - for the verification. I created an action which only works in the development environment to access the verification link as a webpage. So now I can test the whole flow. Also spelling out the wordings and phrases in a selinium test script helps you spot awkward phrases. You might also spot opportunities for A/B testing
Testing in production is really hard. Ideally you have a ‘staging’ version of your site which works almost exactly the same as production but would touch a staging database and have mails or other calls to external systems rerouted to a log or something else.
Again, ideally, your staging database is a reproduction of production with perhaps user emails changed or other identifying information if it’s required by regulations. That way you can safely test new features in the staging system.
I have had to do functional tests in production though and in this case we hacked a coupon code to give the test user everything for free. Be very careful with things like this though, as I said, ideally you don’t test prod, you test a replica of it…