Actually, there is a problem with a small percentage of the people with a lot more spending power than everyone else. In a market economy, each dollar you spend is a like a vote for what is valuable, and what should be produced. This works fine if everyone has relatively equal amounts of dollars to spend - you get something like economic democracy. However, if some people have lots more votes than everyone else, then the things they want get lots more votes. This draws resources (raw materials, equipment, labor, etc) away from producing things for the average person, and results in more production of luxury goods for the oppressive minority.
The result is that things get more expensive for the average person, because resources have been drawn away to serve the wealthy. In addition, you traditionally assume people who are well paid are more valuable to the economy. This is not true when there is large spending inequality. Those serving the wealthy (limo drivers, personal assistants, etc) are paid more not because they are more valuable to the economy, but because the wealthy can spend more. Those serving the poor are paid less because the poor simply have less money to spend. The result is an economy that more and more rewards serving the few people that least need additional people catering to them.
You might say there is a "hierarchy of value" (extrapolated from Maslow)...
There are some things everyone must have so they can survive: food, oxygen, warmth, etc. Basic stuff.
Once they've paid for their most basic stuff, then they spend their money on other stuff.
The more money they have, the more "esoteric" their later spending becomes.
Pearl divers in general are probably not your richest people in the world - so what do they do if they have a pearl? Or instead of asking that question, let's ask what would they do if you gave them a Picasso or Rembrandt? Pretty much the same thing they would do with the pearls - sell it to rich people who would actually use it, and then spend the money on more basic necessities, like running water and electricity.
So why aren't there more of these poor cliff divers producing food, building homes, laying pipes, or building power plants for themselves to use? Why do they instead dive for pearls? The answer is that in the economy in which they live, wealth has been concentrated. When wealth is concentrated, it becomes more profitable to serve the rich than it is to serve people of your own class - thus poverty continues on in the lower classes, and on, and on, and on.
This is capitalism's own economic calculation problem.