Plural marriages - Who are they hurting?

I just read Confessions of a Libertarian Polygamist and found it fascinating. I have talked with several people in the past year that either come from plural marriages or are polyamorous themselves. In fact, I am very surprised at the number of folks I have met and the vast numbers that seem to exist.
The thing is, it doesn't bother me in the least. In fact, as long as all parties involved are consenting adults, what business is it of anyone else? I find it difficult to maintain a relationship with one wife, so I am not sure I would have much luck in a relationship with multiple loves. In any event, I think adults should be allowed to be with whoever they love - and if that means more than one person - more power to them. The world needs more loving families.