Paul Ryan may be forced to leave speakership by summer: Report