This is all something I learned the hard way as I looked out across the White Mountains State Park from a 4,700ft high waterfall. By the time I had reached this elevation, the weather had changed from a pleasant 70°F and clear skies, to about 20°F with storm clouds blanketing the summit. I was nearly a mile away from the summit when swaths of people began coming down the mountainside. I continued on for a while, but as more and more people passed me going down, not one had said they made it to the summit, and even more said that it was way too windy to keep going. So I made the call to turn around and follow them all down.
I found out later that night that at the time I had turned around, the summit conditions were 24°F with 55-75mph winds, and gusts of up to 95mph. Needless to say, I was glad I turned around.