AoS is one of the few geeky shows my wife will sit down and watch with me. She loves it. I don't think it's fair to look at the ratings in the beginning and judge the show on it's ratings compared to now. Of COURSE it's ratings were going to be sky-high. All the hype leading up to it, and the fact that it's tied into the Avengers movies, everyone who was a fan of the movies is going to tune in at first, then all the sudden it's "Waaaait a minute... Where's Captain America? Where's Iron Man? You mean this is a season-long TV show and my tiny little attention span isn't going be fed immediate super-heroes, explosions and conclusions to storylines within the hour?" And viewers start dropping.
As far as the characters, the only non-agent is Skye, and even she's training to become one. That makes sense. It introduces the audience, through Skye's prospective, to SHIELD and how it works.
Fitz and Simmons, while not FIELD Agents, are still a really important part of the show. Especially when you look at Joss' formula for past shows. It's all about the ensemble, and sometimes it takes a bit for the show to get it's momentum. Angel, for example. Terrible first season, really awesome by the time it ended. But you look at Buffy, Angel, Firefly, Dollhouse, they all have a core group of characters that fit certain little niches. Willow, sometimes Oz, sometimes Wesley, Fred, Simon, Book, Topher... Fitz and Simmons. They might not seem important at first, maybe even in the way, but they're the heart and soul of the group that help mediate and keep everyone together. They end up being the Samwise Gamgees of the group. The conscience.
So, I think being linked to the movies is keeping it bogged down. They want to keep enough links to the superheroes to keep people coming back for pieces of that, but the show isn't ABOUT the superheroes. It's about SHIELD and the men and women who are dealing with a world full of things that they don't understand and a world that's changing uncontrollably. If you're expecting superheroes, you're watching the wrong show. This is specifically NOT about the superheroes, but the non-super heroes and the world around the superheroes. I think once you stop looking at it as an Avengers Mini-Movie and giving it all those expectations, the show stops being so disappointing.
but that's just my opinion.