Time between infusions

I'm new to loose leaf and I'm wondering- does how long you wait in between infusions when steeping the same leaves multiple times matter? For instance this morning I got very distracted and came back after 20 minutes for infusion two. (It seemed fine but I don't know what I'm doing). Is there a point at which it's best to just toss your leaves or can tou come back for that next infusion any time (within reason of course)