The First Amendment protects both the right to associate, either formally through an organization or informally, and the right not to associate, which means the government cannot force people to join a group they do not wish to join.
The United States Constitution, What It Says, What It Means, A Hip Pocket Guide