'Screw up' is a common English idiom that has multiple meanings. The most common usage is to mean 'make a mistake', or 'mess up'. It's uncertain how it came to have this meaning, although we have some clues. If you are an ESL speaker, the large number if different meanings is actually good news, as even native English speakers need to rely on context to decode the correct meaning.
1. verb To make a mistake. To mess up. "You really screwed up there."
2. noun A person who often makes mistakes. Often hyphenated. "Bob is a real screw-up."
3. noun An undertaking or event that did not work or is doomed to failure. Often hyphenated."Well, that was a royal screw-up"
4. adjective To be in error, to be wrong, to be broken or messed up. Often hyphenated. "That's one screwed-up car you have there"
To understand how these and other various usages of 'screw up' came about, it is important to realize that in centuries past a screw was not just a nail with threads. Many people were more familiar with the large screws that were used to tighten down and raise up presses -- whether a printing press or a grape press -- and corkscrews. 'Screwing something up' was a literal synonym for 'raising something up'. It has been used figuratively since at least the late 1800s. It is likely that usages such as 'he screwed up his tenant's rent' helped build the negative connotation that 'screw up' has today. But we still use it in the older sense:
5. verb To increase. "He screwed up his courage and marched into the room." (To 'Macbeth I.vii|screw up one's courage]' is nearly the only way in which this sense is used, although you may occasionally hear about someone screwing up his resolve or determination)
'Screw' can also be a synonym for 'twist'. This probably also contributed to the meaning of 'screwed up' to mean 'messed up', and 'screw up' is still used in the sense of 'twist', usually in regards to facial expression.
6. verb To contort. "He screwed up his face in rage." / "He had to screw up his eyes as he looked into the bright light"
And finally, the literal use of 'screw up' is still in use, although less common that the idiomatic.
7. verb To attach, to fasten to a high place. "I screwed up the mirror."
8. verb To raise up by means of the screw. "Screw up the press!"
A number of noders have suggested a link between f*** up and screw up. It is certainly worth mentioning that screw up does indeed have nearly the same meaning, and shares a syllable, with a popular obscenity. Screw up is used as an escape word, and may be viewed as a mild swear word, but etymologically speaking, using the word 'screw' for sex probably did not directly influence the meaning of the phrase 'screw up' (although it may have influenced the popularity!) I could be wrong, of course. I cannot find any information on when the idiom 'f*** up' first came into use.