I agree. I think it was all part of the Illuminati's plan to get Cap out of the picture. Using the "gems" was simply an easy way to get the Time Gem out of Cap's possession and a test to see how he would react. When he reacted negatively to their plan, they mind wiped him and sent him on his way.
In that case, it would be more like the original members of the group's plan. If Tony, Reed, Namor, et al. didn't trust Steve with the gems, they probably don't trust T'Challa and Hank with them, either.
Otherwise, the gems would've shown up again by now.