The numbers vary from one data set to another, but generally speaking, homeschooled kids score well above average in comparison with public school students. A common reaction to those data is the claim: "Of course homeschoolers do better, because they're a small privileged group with highly educated and devoted parents." It's exactly the kids who would've done great in school anyway who choose to homeschool, or so this argument goes. The home educated kids who win the national spelling bee or geography tournament tend to reinforce this notion that homeschoolers are selected from among the bright kids with the academics-obsessed parents.
The thing is, dissatisfaction with the academics at school is not cited as the most important reason for homeschooling by most families. From 2003, we have these survey data:

(That first footnote is to clarify that "environment" included safety, drugs, and/or negative peer pressure.)
Note that while 16% cite unsatisfactory academic instruction at school, perhaps implying that their child is bored and unchallenged, 14% cite difficulties which may include learning disorders, ADHD, serious anxiety, or other factors that would tend to decrease standardized test scores.
Also, many of the people who feel homeschooling is undertaken by the privileged (and therefore is unfair and un-leftist) also believe that Bible-thumping backwoods homeschoolers are abusing their children through skewed and inadequate education. Well, you can't have it both ways. If the non-college-educated fundies in Missouri or Utah are guilty of criminal neglect for homeschooling, then wouldn't that tend to pull down those homeschool statistics? Mind you, I don't agree at all with these stereotypes, but they're out there. Since religion is cited as a reason for homeschooling about twice as often as dissatisfaction with academics, then if anything, according to some, we ought to expect below average test scores. (By the way, in this study of 2009 data, homeschoolers tested in the 86th percentile in science.)
In my college town community, many people homeschool as an extension of attachment parenting. We like spending more time with our kids, knowing what's happening in their lives in more detail, and reducing bustle and stress. (I once told a friend that getting my kids dressed, fed, cleaned up, and packed for school before the bus came would literally be harder than homeschooling-- and I was not joking.) Beyond attachment, the reasons are extremely varied. One family keeps strictly kosher, one has boys whose nut allergies are potentially life-threatening, one has a son who's dyslexic, one has a daughter who faced bullying in school, one doesn't like the pressure her daughter experienced to be "girly". Some kids certainly have interests they couldn't pursue in school (or not as much), such as music, theater, or chemistry (at age 11). Anya's major interest is the breeding of animals and genetics. But of course, interests that wouldn't be met in school would also not be on the test. So, when parents cite dissatisfaction with academics, that may be because there isn't enough emphasis on their child's particular area of interest-- but that interest is not likely to improve test scores.
I think it comes down to the most obvious thing: One-on-one tutoring is a lot more effective than one adult teaching 25 kids in what is (for many kids) a hostile environment. Just remember that every homeschooler represents a smaller child-to-teacher ratio in some classroom somewhere, so homeschoolers and schoolers are not natural enemies here.
As for how privileged homeschoolers are, here is some more 2009 data:
- The median income for home-educating families ($75,000 to $79,999) was similar to all married-couple families nationwide with one or more related children under age 18 (median income $74,049 in 2006 dollars; or roughly 78,490 in 2008 dollars).
- Homeschool parents have more formal education than parents in the general population; 66.3% of the fathers and 62.5% of the mothers had a college degree (i.e., bachelor’s degree) or a higher educational attainment. In 2007, 29.5% of all adult males nationwide ages 25 and over had finished college and 28.0% of females had done so.
- Almost all homeschool students (97.9%) are in married couple families. Most home school mothers (81%) do not participate in the labor force; almost all home school fathers (97.6%) do work for pay.
- The median amount of money spent annually on educational materials is about $400 to $599 per home-educated student.
This does suggest that homeschooling families are of higher socio-economic status. Homeschooling parents are more than twice as likely to have college degrees as the rest of the adult population. However, that being said, in the 2009 study, even when neither parent had a college degree, kids still scored in the 83rd percentile on "core" subjects (reading, language, math). Families where both parents had a college degree had kids scoring in the 90th percentile. There's a difference there, but parental education doesn't come close to explaining the gap between homeschoolers and schoolers.
Regarding income, I'm surprised (maybe even a little incredulous) that there's little or no income difference between schooling and homeschooling families. Though I do live in a college town where many folks are degreed, most homeschooling families that I know are scrimping to get by, and in quite a few cases live pretty close to the edge. Second-hand clothes and toys, CSA's and food co-ops, and heavy coupon usage are the norm. And most of the moms I know do some form of work on the side, it's just that most of it's under the table, and would therefore not be reported in a survey.
It's odd to me that what I'm saying here is often taken as highly controversial. I'm saying that parents are probably better at teaching reading and arithmetic to their kids than are teachers who are 1) responsible for 25 kids and 2) totally hamstrung by rules, regulations, and the current fads. Feels to me like I'm stating the obvious. And yes, maybe parents without high school diplomas have a harder time teaching their kids to read, but the thing is that, beyond the sort of phonics you see on Sesame Street ("S" says Ssssssss)... I didn't teach Anya to read at all. She taught herself using the Webkinz website, comic strips, and Bunnicula books.
Homeschoolers actually tend to be non-judgemental, because we ourselves get judged so often. But our presence in the world does tend to raise some questions about why certain schools are failing, or why schools are failing certain kinds of students.