The only true test is in an actual combative situation - any other environment, regardless of intensity, will still only provide a speculative result.
I do not think this is true at all. It's not at all difficult to know when things are martially ineffective
. Aikido practice is highly stylized. So the first sign of martial ineffectiveness is when the folks in the dojo can't even make their waza work under the controlled circumstances of practice.
Irimi is at the heart of all martial application. If you go to a dojo and no one can enter without you hitting them, the practice is ineffective. That's my first test... I frequently arrive at dojos to teach and find that not a single student can pull off an irimi when I attack. That's because I REALLY attack. at the majority of the dojos I see around, the students are not really trying to strike their partners. If everyone trains that way day after day, they think they know how to do things they really cannot do. As Frank Doran Sensei says, the "entry" is everything, everything else is just icing on the cake.
I think that the "entry" is the most neglected aspect of Aikido training. I sell a lot of Aikido DVD's. I have a set I call the Principles of Aiki set. Vol. 2 is just on "Entries". I sell fewer of those than the others. I am convinced that this is because people see the title and say to themselves "I know how to do that..."
Anyway, it's a shock when a bunch of third or fourth dans, or even worse, someone running a dojo, finds that they can't do an entry. They can know 500 techniques and without effective irimi, it's just 500 techniques they cannot do.
The second thing one can spot at a dojo at which the practice is clearly martially ineffective is closely related to the above. Can the students at the dojo strike? With speed, with power? If not, then the practice is being done at unrealistically slow speed. People will not be able to adjust when it gets fast and hard.
What does the "intention" feel like during practice. Once again, you can look at the folks in many dojos and see that they have no projection, no forward intention. You can stand in front of them and feel nothing. They have no idea how to organize a strong forward flow of attention. If you attack them fast, or God forbid, with unexpected timing, they are never ready. You can stand in front of someone like this and know you will hit them before you even start.
One of my students gave me a book on the theory of limits as applied to business. While being over my head math-wise after about three chapters, I got the gist of it. It changed my thinking about how we teach our art. The theory of limits says that in any complex system, like a factory (and Aikido is also a complex system of body / mind skills), one needs to analyze the various elements that go into producing the output of the factory and decide which one is the "limiting factor". You can throw all sorts of money and resources into that factory and have no increase in the production whatever if you don't devote them to improving the "limiting factor".
So, in my opinion, most Aikido practice is done without any regard to this idea. People are studying a wide range of techniques, empty hand and weapons, putting all sorts of time and money into their training with almost no increase in actual skill from year to year because they have not addressed the limiting factor in their Aikido.
For the majority of the folks I see training, the limiting factor is the lack of ability or willingness to train with attacks which have speed and power. Strikes have no body integration and hence no actual power. Grabs tend to be "strong" in a way that is totally ineffective. A grab should be designed to effect the partner's balance and his ability to respond. Turning your partner's hand purple by grabbing really hard has no martial effectiveness whatever and is probably making you tight in a way that limits your ability to move freely.
So collectively, I would put all of this under the label of "attacks". Problems with the "attack" is the limiting factor for most Aikido folks. There is simply no possible way for someone to get to any level beyond the rudimentary without addressing this issue. Period. 50% of ones training is in the role of "uke". All sorts of attention is put on the ability to take the fall, very little is put on the actual attack.
Now, that said, fixing this issue is still no guarantee of "martial effectiveness" outside the dojo. But the idea that only combat will tell you anything simply isn't the case. Physical conflict runs through a whole range from a drunk guy shoving you at a bar to two or three fellows with guns confronting you on the street. There innumerable stories of folks with only moderate skills, developed in their dojo environments, using their Aikido "effectively" for self defense on the street. The reason for this is that most attackers out in the real world are not formally trained in anything. Many are simply incompetent. Dangerous perhaps, but not very sophisticated.
Combat is all out, life or death. Most folks will never have to use their Aikido in combat. That doesn't mean that one can't train for martial effectiveness. Do you want to know whether you are "martially effective"? Go up to your local mixed martial arts gym and see about applying what you've worked on in the dojo. Personally, I don't actually care about this issue, but young men often wonder if their stuff "works" and this is a good way to find out. The uchi deshi used to wonder the same thing... they'd go out to the local bars and get in fights, often with the soldiers from the occupation. That's a good way to find out of you can do your Aikido against folks who have no formal training. Of course you might get injured, killed or just plain arrested doing this, but it will tell you something.
Anyway, my feeling is that people need to fix how they train in the dojo and get it to the point at which it actually works within the stylized framework of Aikido itself before they need to start worrying about "combat application" or "martial effectiveness". These discussions are off mark most of the time, I think. Find the "limiting factor" in your training and fix it. Then find the new "limiting factor" and fix that. Progress will result and eventually you will be good at what you do. Then, if you want to experiment with non-traditional applications, go ahead. Folks who worry about this too early in their training typically do not get very good.