Nowadays people don't go to some physical location and invest their money there. A lot of it is done over the internet, by phone, and through the Post Office.
Many people who invest their money don't even live in the same place where their money is invested. And they are not in a good position to investigate and check things out.
From what I've read about it, Mr. Madoff encouraged his investors to start with a small investment. And if they liked the results, then they could invest more of their money later on.
Of course, he made sure that the results on that small investment were always likable. And virtually all of his investors ended up investing a lot more money, some invested as much as 100% of their money.
People don't normally investigate a lot some well known guy with a very good reputation, when they are investing only a small amount of money with him. And when they get very likable results from him later on, then they know from their own 'experience' that he is a good money manager.
Good results tend to inspire trust. He always 'produced' good results. And that's why nobody felt like investigating him much.
People usually start investigating when they get some bad results. Which is why everybody is investigating Mr. Madoff now.