This thought arises from what I saw happen at my question. 2 answers in succession were given, then subsequently deleted by the posters themselves. I missed parts of it, but I presume comments that raised valid questions (lack of references etc.,) led to it. This further led to the current "answer" by another user which repeats one of the deleted answers. Will it not be easier (for readability and documenting purposes) if we retained the "deleted" answers, but just dimmed them out or have them collapsed with a note, like comments marked spam on youtube? It wont lead to similar confusing situations arising again. It also means that new users would then know what answers were already given and the reason why they were "bad" answers.
I cannot speak for any decisions that were made, but based on my experience, I think this was the case:
If the answers were simply poor quality ones, they might have actually been retained, and allowed to accrue downvotes for their quality. However, they were not answers for various reasons.
The first answer was a link-only one; had it had more substance, it could have potentially been a comment, but as such wasn't worth anything beyond putting the topic of the question into Wikipedia, and, in addition, "link only" answers are discouraged on Stack Exchange for reasons of "link rot" (disappearing links due to external sites moving domains, etc.).
The second and third answers, contain exaggerated or blatant misinformation, which the second answer's author was asked to substantiate with reputable sources (and so far, the third answer's author has not, either, and a message has been attached to his answer, stating that). After prompts were made from a moderator to do this, the second user elected to self-delete his answer (thanks Mad Sci, for the correction on that). This is for the best, though, as were someone to have been searching on Google for "itching", a hit on that question would have brought them to those answers, and not provided them the proper information for which they were seeking.
I know that's specific to this case, and you were speaking more in generalities, but I think these are some of the reasons behind not leaving such a trail for general users (as Mad Scientist has mentioned above, with elevated privileges, these are in fact visible).
If you have 2k or more reputation, you'll be able to see deleted answers. So something similar to what you requested is already implemented, but it is tied to a minimum reputation level.
When posts get deleted, most of the time it is for a good reason. They might have been spam, non-answers or bad answers that the author self-deleted.