Private mortgage lenders with the backing of Wall Street’s biggest names are facing an unsettling reality: Their breakneck growth has made them increasingly vulnerable.