I've flown in some heavy winds, both at my current position, and while flying as an instructor. I have some interesting stories about wind, often in clear conditions.
Basically, wind can be as serious a weather phenomena as low ceilings and fog when it comes to flying. I'm not going to judge you or your instructor, since I don't like being "Monday Morning Quarterbacked", as I'm sure you don't either. But, I can say, when flying the Saab with winds reported 21 gusting to 30, we find that to be enough of a concern that we at 15 knots to our Vref for a Vfa (final approach) speed, just in case of gusts or shear.
One time in particular, I was the PF and landing on 31C at MDW. Winds were reported around 240 at 30G45 or something to that nature. However, asking for a wind report from tower, we got "Winds 230 at 20." Ok, not to bad eh? Well at about 30 feet above the runway, we lost 20 knots, and the plane came down hard and bounced right back up, and we did a go around. Due to the winds hitting the hangars and fence surrounding MDW, the winds dropped off greatly down low, and so did we.
The same day, we had a flow control delay while at DSM, so we did a 180 turn to hold at the pad waiting for departure. We wanted to turn into the wind, so as not to blow all our exhaust right back into the tailpipe (tailpipe hot) and into our bleed system. After completing the 180 turn, the winds pushed us backwards, and castered our nose wheel. We didn't know it at first, but a few minutes later when attempting to taxi, we started going right, with our nosewheel stuck at the 90 degree position. Not good.
Departing MDW the other night, there was shear reported on final for 13C. They were using the ILS 4R circle 13C approach, and we were departing on 13C. On initial climbout, we had an amazing amount of turbulence between 1000 feet and 2500 feet. This was due to the great shear between the winds on the ground (slowed by friction off the buildings, and disturbed by them as well) and the high winds 50 knots plus at 3000 feet. Our ground track heading 270 was pushing us towards Wisconsin. Winds on the ground were reported 15G20.
I have plenty more stories about the effects of winds that don't seem too strong (in the 15G25 knot range), and even more about winds that ARE strong (in the 30G50 knot range).
You have to understand that winds have a serious effect on flight, as does ice, snow, rain, fog, or any other of the more obvious weather phenomena. We often don't consider wind to be something to be a determining factor in cancelling a flight, particularly when the weather is clear. Now at the airlines, we won't cancel for wind, unless it's like a hurricane! I've landed in winds reported gusting to 48 knots. But, for those flying smaller aircraft, winds up to 30 knots can be detrimental to the operation.
Consider the aircraft you are flying, it's limitations, and your location. Do you think it's safe to fly? Don't worry... the more conservative approach in making the go/no-go decision in winds isn't the easy way out. It's the hard way to go. Because then you will be judged, but you need to have enough confidence in yourself to say "I'm not flying in that weather."