Why are estimates so difficult in software development?

Software estimation is difficult, and what some people try to do with software estimation isn't even theoretically possible. Upper management, lower management, customers, and some developers don't seem to understand why estimation is so hard. People who don't understand software estimation's inherent difficulties can play an unwitting role in making estimation even harder than it already is.
— Steve McConnell in Rapid Development

1. At the start of a project, uncertainty is very high

All of the other reasons listed below are really just expansions of this one principle. Research indicates that estimating a project in its early stages yields estimates that may be off by as much as a factor of four. There is just too much "unknown" to give a reliable estimate. The problem is so widespread that it birthed the #NoEstimates movement. 

2.Every project depends upon third-party software

This third-party software can change at any time, and the the developer has no control over this. Developers regularly make use of open-source, shared libaries of code which are maintained by thousands of developers across the planet. This is an efficient way of developing software (thus avoiding "reinventing the wheel"), but it comes at a cost. Namely, the third-party code can change at any time, and frequently introduces incompatibilities with existing code.

These third-party code changes cannot be predicted, nor can the incompatibilities they introduce. A software feature might have worked perfectly last week, but one of these code changes this week broke it and now requires extensive rebuilds to make it work again. These kinds of unexpected changes can and do increase development time.

Enterprise software must keep these libraries up to date, and this requires constant, ongoing maintenance.

3. Every project is unique

A cabinet maker can easily estimate how long it will take to build a cabinet, because he's made hundreds of them before, and they're all very similar to each other. In software development, every project is a custom work, and is different from everything that came before it (if it weren't, we'd just reuse what we've built before). In other words, it's not repeatable. In this field, the developer is always being asked to do something he's never done before.

4. The larger and more complicated a project is…

… the wider its estimated probability distribution is. This is just another way of saying that the larger a project is, the less certain the completion goal is, and the more likely unexpected difficulties and changes will crop up.

5. Requirements are often in flux

Not only are the tools in flux, but the requirements themselves are often changing. Markets change quickly. Needs are discovered in the process of development. Many large projects morph into something much different than what they started out as. Oftentimes, stakeholders realize that their initial ideas are incorrect, or won't work, or need significant tweaking—and these changing needs don't get discovered until halfway through the project.