"It's been a while for me too." He's already picked up on tells that most of the team he's run into are from either further ahead in time than he is or their missions took them far enough afield that he couldn't keep discreet tabs on them.

"The last time I saw you, the ah, other guy, was stealing a quinjet and leaving Sokovia far behind."

So yes.. he was there. At least he had eyes there.
Someone had to get Fury those helicarriers and point him in the right direction after all.