When I first started running 4e, I had a party of 5. A couple of players dropped out after the first few weeks and from there I ran a 3 player campaign for about a year. It was excellent. Combat was fast and I could really focus on stories specific to the PCs.
In my estimation, the ideal three player party in 4e is a Leader, Striker, and "something else". The "something else" can be any role; in my game it was a Defender. The key is the first two classes. None are required by any stretch; you could run a three-Witch party and have great success. Still, I think ideally you have the Leader for healing, the Striker to give that extra oomph in combat, and one other role to compliment the two.
There are other benefits from running with small parties in 4e, besides the obvious one (i.e. much quicker combat). A lot of people have a hard time putting a large group together. It is a lot easier to find 3 players than 5 or 6. 4e is very easy to balance around different numbers of players; you just adjust XP and there you go. Finally, it allows you to focus more on each individual PC's goals, personality, and backstory; you might find that this improves the roleplay at your table.
If I ran another 4e campaign, I would be looking at going with 3 players, 4 players max. The game doesn't suffer from fewer players, it actually improves.
Do you have any experience running 4e games with smaller parties?