Solid tantalum capacitors. The loose capacitors at left are 10 uF; the caps fastened to the tape at right are 1 uF. Note the penny for size comparison.
First, let's talk a bit about how tantalums work. Tantalum capacitors have something in common with electrolytics: in both cases, the capacitor dielectric consists of the oxide of the metal that one of the electrodes is made of. In the case of the electrolytic capacitor, it's a layer of aluminum oxide that forms when the electrolyte reacts with the aluminum anode under an electric charge. The electrolyte is necessary to maintain the oxide layer. When it eventually evaporates or leaks out of the capacitor's container, the oxide layer breaks down and the capacitor loses capacitance.
Most tantalums used to be made the same way; these were called "wet slug" types. The ones that fail inside '70s ARP synths are wet slugs. The problem with these is that, because tantalum is pretty unreactive compared to aluminum, strong acids have to be used as electrolytes. These eventually eat up the seals around where the leads penetrate the body, and the electrolyte gradually leaks out. However, at some point, someone figured out that they could use a process where the tantalum "slug" is dipped in an electrolyte at the factory before the capacitor is assembled, and connected to a voltage to form the tantalum oxide dielectric layer. Then, the slug is removed, dried out, and assembled into a capacitor; the electrolyte is used only at the factory. Because tantalum oxide is more stable under electric charge, the electrolyte isn't necessary to maintain it. These are the "solid tantalum" or "dry slug" types. There are several different designs used in solid tantalums, but they all use the same basic idea -- a tantalum slug forms the anode; a layer of tantalum oxide built up on it is the dielectric, and some material (often manganese dioxide) is bound onto the outside of the oxide layer to form the cathode. Thus there is no electrolyte inside the assembled capacitor; since there is nothing to leak out, the capacitor will not degrade or lose value.
Cutaway drawing of a surface-mount solid tantalum capacitor. From an NEC data sheet.
As it turns out, tantalum oxide has a very high dielectric constant. This means that a tantalum capacitor can use a very thin layer of oxide, only a few microns. The basic rule of capacitors is that the capacitance value is directly proportional to the surface area and inversely proportional to how far apart the electrodes are; since the electrodes in a tantalum cap are separated only by the thin oxide layer, a tantalum cap can pack a lot of capacitance into a small volume.
The problem with having an exceedingly thin dielectric layer is that it can't withstand high voltages; it doesn't have enough insulation value. Thus one of the first rules for dealing with tantalum caps: do not expose them to excessive voltages. Most references I've seen recommend that tantalums be derated to 50% of the rated voltage. Some say to go down to 30% when the tantalum is used in a low-impedence, high-power circuit. Inexpensive tantalums are usually rated for 25-50V, with higher voltage ratings becoming harder to obtain above 10 uF.
The second thing about tantalums is something that they are notorious for: exploding. Why does this happen? Two words: reverse voltage. The polarity of a tantalum cap must be respected. Any reverse voltage breaks down the dielectric layer. It reduces the oxide back to metallic tantalum, which then forms a short-circuit path; if the circuit the capacitor is in can supply a lot of current, the short path heats up rapidly and the capacitor goes boom. The third point is related: ripple current. Large amounts of ripple have a similar effect: they create localized heat spots in the dielectric, which breaks down at those spots and a short circuit results. So it is generally best to avoid using tantalum caps in applications where they will be exposed to high ripple current. (Nonetheless, some power supply makers use tantalums as filter caps in their supplies, and it works. How they get this to work, I don't know.)
So why use tantalums? For one thing, they are small and light, much smaller than electrolytic caps of the same rating. Second, solid tantalums can't leak because they contain no electrolyte. "So what", you might say. In a synthesizer, does it matter if the capacitors are a few grams lighter? Probably not. No electrolyte is an advantage, but many synths are loaded with electrolytic caps and they seldom leak, and even if they do, it's usually not that big a deal.
Well, there's another, very good reason to prefer a tantalum capacitor over an electrolytic. Consider:
This is an interface circuit that I built for a home automation system. It's basically a 7555-based monostable timer circuit that, when activated, holds a relay open for a fixed amount of time. The little yellow blob at bottom center, just above the red connector with the blue and orange wires going to it, is a .68 uF tantalum capacitor. When I prototyped this circuit, I used an electrolytic capacitor of the same rated value. When I fired it up to check it out, the actual value of the cap as determined by the circuit's time constant worked out to about .35 uF -- unsatisfactory, because the timer needed to hold for 600 milliseconds and I was only getting about half of that. Electrolytic capacitors are not noted for being precision devices; the .35 actual that I got from the cap marked at .68 is within the typical tolerance for electrolytic caps, which are often specified as being -50%, +100%. And, electrolytics will lose value over time as the electrolyte evaporates; it isn't unusual to pull one out of a circuit after several years and find that it is only operating at 10% of its marked value.
Standard tantalums, on the other hand, are specified at plus or minus 10%, and you can get tighter tolerances at a somewhat higher price point. And solid tantalums will retain their operating value over a long period of time. (Wet-slug tantalums are still made for special applications, but don't waste your time with them.) The advantages for synthesizer circuits should be obvious: when the calibration of, say, a VCO or a VCF depends on a capacitor circuit, tantalums will make the circuit closer to the center of the calibration range at aseembly, and will retain their value over time, reducing the need for recalibration. That's why I went with a tantalum when I built the board above; the .68 uF marked cap got me as close to the 600 millisecond time value as I could easily measure (50 ms or so) without trimming.
The one other thing about tantalums: there is no such thing as a non-polarized tantalum cap. So keeping in mind that you need to respect the polarity, a bit of care is required in design. Do that right, though, and you'll have a more stable and reliable circuit. After all, tantalums are considered reliable enough by the aerospace industry that they are heavily used in aviation and spacecraft. So don't be afraid of the tantalum.