De,
I hope you're young, because farming and livestock are extremely hard work and one person alone, a farm does not make. It can be a financial sink hole. Then, people have done it and you're really enjoying it, which is important.
You appear to have considerable resources, which is good. I will tell you in my retirement experience, the cash goes three times faster than you think it will, so I would hold out in the other direction and work as long as possible, unless you're really well off. Maybe stay in Virginia while your son's going to college, for example. I think you will find being so isolated from family difficult. I'm always hearing stories of parents moving to be closer to their kids.
It's your decision to go survivalist, but I'd offer that economic conditions have changed since you started the endeavor and, thankfully, the Constitution dumped loser Obama. With Trump bringing manufacturing back and lowering taxes, it seems the economy should slowly turn around, not collapse. That said, there are a number of natural disasters that could take us down, not to mention enemies. So, there's no guarantees on anything. But I liked your approach of having a foot in both places.
Just a thought for your consideration...