We bought our first home in 2002 for $216K, and sold it for about $310K 5 years later. The first place was 1600sf, and after the 2nd child, was getting tight. We took $90K to put down on my current home, which was 20% down on the $450K house (no PMI). Ever since I was a kid, I had parents and grandparents follow this pattern and become wealthy. Because if I could write a check for 20%, that's not too much house...
That last transaction was summer 2007, and within a year our county tax assessment came in at $260k, down from the $450K purchase price
In hindsight I could have been $250K ahead if I magically knew to rent for 18 months in between homes - but of course I don't have magical powers.
Trust me, none of us do. I'm in the same boat, and I didn't have the luxury of flipping a first home to get that down payment. The first condo I bought (at 25) was literally all the money I had saved up from busting my ass from when I was 14 up until that point. It was pretty devastating to fast forward 5 years and watch that $135,000 evaporate into $44,000 (market value)... and know I still had another 26'ish years of payments to make.
A lot of my friends in the same boat just walked on their condo/home, destroyed their credit, waited 5 years, then bought a new home. In hindsight, I should have done that. I'd be ahead of where I am now.
I can't look at it that way though. I made a decision and I'm past the walk away point. I also can't kick myself for not waiting 14-18 months. I did what I thought was responsible, I got dealt a bad hand... time to re-ante up and move forward.
Bigger picture, what I'm saying... is if you keep making payments and paying that principal down. You're not going to cash out with only $20-30k. The only way that's going to happen is if you start going crazy and borrowing against the equity every time you pay it down enough to do so.
Whatever you sell at is money towards retirement. If you pay the house completely down and sell for $200k (let's say that's a worst case), don't look at it as a $250k loss... because you're going to have $180,000 or so when you walk away. View it as $180,000 towards retirement.
Also, you might be able to sell for $450k (you never know with the long'ish term housing market) and then all of a sudden you're walking away with all the money you need for retirement (combined with your other vehicles).
It took me a really long time to come around to this way of thinking. I kept saying "well, I basically lost $100,000" ($135,000 down to $44,000) on that investment.
That's not accurate at all though, it's not a loss until I sell (or forfeit by not making payments).
If you hang onto the home and pay it down, that $250,000 liability turns into a $200,000 liability, down to $100,000 and then you have it completely paid out and you're looking at a nice profit when you sell. Even if you sell for $200,000... if you've paid the house down that's not a $250,000 loss, it's $180,000 (after fees) towards retirement.
Don't factor in what you initially put down... that's gone and never returning. Factor what you can expect to get out of it when you sell. (assuming you keep paying down principal)
---
Not saying you don't need to readjust your 401k or IRA, just saying that you need to factor in the future sale of your home as well. It's a very important and potentially large piece of your retirement puzzle in 20 years.