Many of the holes in the IPA chart exist because of either two reasons:
A. This sound exists in no human language because it is nearly impossible to produce. An example would be a pharyngeal stop, since it would involve retracting the tongue root to touch the back of the throat.
B. This sound is literally impossible to make, a good example being a voiced glottal stop. Another classic example is a bilabial lateral. This is usually due to the fact that the place of articulation conflicts with the manner of articulation. These impossible sounds are represented as grey blocks on the IPA chart.