It all has to do with the engine. And economics.
The market was small for the 3 engine wide body and probably at some point Boeing realized Lockheed and McDoug were killing each other with their DC-10 and L-1011. Neither company made much money on those airplanes and some articles said that the only way McDoug stimulated DC-10 sales was by doing away with the stretch -8, a decision they later rued as it became a great freighter. the 8-73 with the big engines could haul the freight.
As for other designs, Harry Stonecipher, once head of McDoug, stood before the press at Paris and said that were not McDoug in the commercial airplane business, it would GET into the business because there was room for innovative designs. One was the MD-12. Of course, it never got off paper like many other designs and proposals.
FWIW, the 757 began originally as the 727-
300 and was going to use the JT8D-200 series engines. It was offered to United and American (IIRC) and both turned it down. It was not that much of an improvement.
This is the MD-12
And this is the early proposal of the 757 with the t-tail showing its 727 heritage.