Where are all the butterflies?

I think my butterfly garden has turned into a hummingbird, gold finch and bee garden!! Actually, I did see two swallowtails today. A Tiger Swallowtail that was probably the same one I saw the evening before in front of the house. And a dark swallowtail, that I think may have been a Pipevine swallowtail. It seems like there should be more by this time of summer. Maybe more land around here has been destroyed to build more housing so now they have even less habitat areas left.