The advice I've always heard is that if you're writing a surname history, it's "okay" to self-publish as long as you cite thoroughly using BGS standards and have other experienced genealogists go over your work and writing first. Otherwise, and especially if you're writing a local history or more broadly thematic work, you should try to get a genealogical society or even a local history association to be your official publisher, even if you go the POD route.
I have been working on a genealogical book for about a year now. It's about a specific region that has not be written about from a genealogical perspective yet. In order to make it successful, these are some of the things I've considered:
1) There is no due date. Do it right. As every genealogist knows, you're constantly getting new information and revising things. I don't expect to publish for at least another two years at the earliest. Lots of additional research trips to do!
2) Involve *all* stakeholders. In my case, it means including the other major players in the region's genealogical scene, so to speak. I've also worked with local archivists, professors, the diocese (to get religious context), etc. Self-publishing sometimes gets used as an excuse to be lazy with your research, which I actively avoid.
3) Work through a historical society. It gives weight to the book from a legitimacy perspective and provides you with additional marketing and design resources.
What's interesting is that we *do* plan on using a POD service for the actual printing, but the historical society will be the noted publisher. I have most of the control, but I get to use their name and reputation for my benefit. That's a good compromise that a lot of authors seem to go with, and it makes sense to me.
For me, fully self-publishing was never even an option I considered. I wanted to write a book that would be indisputably "valid" in academic contexts. I also really, *really* wanted critical feedback from scholars.