Doesn't say anything at all about retraction or extension.
If you look under §23.345(C, D)
(c) In determining external loads on
the airplane as a whole, thrust, slipstream,
and pitching acceleration may
be assumed to be zero.
(d) The flaps, their operating mechanism,
and their supporting structures,
must be designed to withstand the conditions
prescribed in paragraph (a) of
this section. In addition, with the flaps
fully extended at VF, the following conditions,
taken separately, must be accounted
for:
(1) A head-on gust having a velocity
of 25 feet per second (EAS), combined
with propeller slipstream corresponding
to 75 percent of maximum
continuous power; and
(2) The effects of propeller slipstream
corresponding to maximum takeoff
power.
It would appear, that you are correct in that, Vfe is a limit specified to make sure that the Design limits at Vf are not exceeded. For use, flap retract is bugged as Vfri, others i have seen Vclean, of course this speed is weight variable, and not a limitation.