Affirmative action does benefit members of minorities, but it benefits the greater majority more. In exchange for the tiny, trivial amount of money spent sending a member of a minority to school, or for the miniature average impact on business due to affirmative action-related hiring practices, the majority members of the same body (students, employees) are given a chance to interact with someone that they otherwise might never encounter on an equal level.
To me, this is the only reliable argument in favor of affirmative action, simply because it is impossible to quantify the results of prejudice and discrimination. Just having a particular surname is enough to uncover the prejudices of others even if one is not a member of a given ethnic group. It is literally better for everyone if more members of the majority encounter more minority members. The gain is even larger if they are encouraged to work together, which can (potentially) teach both of them that such a thing is possible.
Affirmative action is about more than just "making up for slavery" (as if you ever could) by getting someone a job. Monocultures are fragile, and thus segregation hurts everyone whether it's government-mandated or self-imposed.