birdonmyshoulder* makes a good point.
To Eliot, or at least to his narrator, the dust is something other than one's shadow behind one in the morning, or in front of you in the evening.
Jung described the Shadow as the part of the mind that one could not see as part of oneself, except by placing it onto the world. Jung's Shadow could be a creature of great fear, as it contained the unknown aspect of the self. While Eliot probably did not lift this directly from Jung, I cannot help but see a similarity to Eliot's shadow here.
In the morning one's shadow is behind one, and one is not confronted with it, striding forward purposefully -- even lifted by the shadow behind, and as life progresses, one sees more and more of one's shadow for what it is -- a part of one's self that one cannot know, and that can instill fear.
But Eliot's dust seems a deeper thing. Connecting this to his heap of broken images, dust is not even a broken image. It is the matter out of which these images grow, the soil, but it is parched soil here. Instead of seeing the images which cover the ground, instead of seeing what your mind creates from the ground, you see only a handful of dust, the raw matter of the world, unmediated by your mind.
And the universe unmediated by your senses and interpretations is a terrifying thing. Realizing that there is nothing to know is much more horrible than realizing that you do not know yourself.