I can count on one hand the number of times in my life I have ever payed to play.
What you described was the exact environment we had at all of our outings. Whether 3 or 20 showed up, it was never an issue that we didn't have refs even with new people. We had all age groups and play styles on the field and we all knew why we were there. To play for the sake of playing. And everyone understood, don't wipe and don't cause any safety issues. Simpler times...
If all goes well later today, I'll be employed and actually able to meet up with some fellow NJ players and relive some of that.