Obligatory self-promotion link to a beta postmortem I wrote in 2004:
Most of it is outdated, but a key takeaway is that I could only keep the attention of testers for about a week. So I’d only push out a beta when you have a near-final version. This is for client-side consumer apps though - if you’re doing something SaaS-like that people use regularly, maybe you can involve testers earlier.
I found that only 10% - 15% of my testers were good at filing bug reports & giving feedback. To find testers, whenever someone showed the slightest bit of interest (eg emailing me with an idea for a product or new feature), I’d send them a link to my beta-test signup form. Best to collect a database of interested people as early as possible.
Recently I’ve tried an Adwords campaign to recruit beta testers, I think it was converting at 10% CTR and resulting in a $10 acquisition cost per tester. The landing page wasn’t much different from a LaunchRock email signup page. That product hasn’t gone into beta yet though, so I can’t comment on the quality of the testers. If you’re paying that much per signup though, UserTesting might be a better alternative (and be sure to look in AppSumo for a substantial discount).
For mobile apps, TestFlight is cool, but I’ve had some problems with people (even close friends) freaking out at installing TestFlight on their iPhones, because of the scary Installing Profile dialogs. I’ve been focusing on Android testing first so I don’t use up my limited iOS test device numbers on unreliable testers.
One last tip: I had a lot of luck with giving a discount code to all my beta testers to share with their friends at launch. That got all the beta testers acting as evangelists for the product and kickstarted word of mouth. (Beta testers all got freebies, of course.)