If you're starting out you can use your mobile phone, simply pop in your headphones+mic, open the recording app and start talking.
If you are going to interview one, or maybe two, people then get together in a quiet, non-echo-y room, pop your phone on a soft something (cushion) onto a table point the microphone at yourself and the interviewees, open the Recording app and start talking. You'll have to sit next to each other as the microphone on a phone isn't built to pick up voices all around ... but experiment, you'll work it out for your particular mobile device.
The next level up is to use a dedicated recording device and we use the Zoom H1 Handy Recorder. This has two external microphones so you can sit how you want. It also records in .mp3 and .wav formats.
Whilst the device came with a microSD card we bought the biggest we could find, a 32 GB microSDHC card, as it's not always possible to get the recordings off the device as soon as they are recorded.
Power: single AA battery
2 GB microSD card
The Zoom H1 comes with free download licenses for Steinberg’s Cubase LE music production software and WaveLab LE audio editing software,
Dimensions: 17.02 x 9.65 x 4.57 centimetres (0.20 kg)
Cost: between NZ$120 and NZ$200
Note: our device is considered a little old so you may get better quality, more space, or additional features for the same price, shop around.
We also got the Zoom APH-! accessory pack, it really does make your life a lot easier to have a dedicated cable (USB to micro-USB), a wind (and beer) shield, a tiny tripod, and a wee case to put it into. Add around another NZ$50 for this pack.
And finally we use lapel microphones with as splitter to record off as these give us better sound quality due to the windshield and are closer to the person talking - a must have if, like us, you record in places with any background noise.
And finally, it's also good to monitor the recording by using some headphones, but that only works if you're able to multitask, and no-one can do that, so just take a look at the recorder levels every now and again.. Oh, and batteries, always carry spare batteries and check the battery level BEFORE you start recording as there's nothing like a recorder dying on you halfway through, really puts a crease in your day.
So there ya go - start simple, turn on your phone, open the app and start recording to make your first ever podcast ... or hire a recording studio ;)