Ways in which new species can form

New species are generated by various processes. One way of classifying these processes is by whether they involve hybridization or genome doubling.

A genome can become doubled when a zygote replicates its DNA in preparation for the first cell division, but then fails to divide into two cells. This is is an example of somatic doubling. There are other two mechanisms: nonreduction during meiosis, and polyspermy. The result is an individual with double the usual number of chromosomes.

No change in genome size Genome is doubled
One parental species Ordinary speciation Autopolyploidy
Two parental species Homoploid hybridization Allopolyploidy

Ordinary speciation involves neither hybridization nor genome doubling. This is the most common process and the most studied. When it occurs repeatedly it forms a branching process, and the result is a binary tree which grows as branches at the tips split into two.

Autopolyploidy is the process where an individual belonging to some species doubles its genome. The individual is incompatible with its 'parent' species and can go on to form a new species. There is no hybridization, and the process can still be seen as part of a branching process.

Homoploid hybridization is the process where two individuals belonging to two different species which (by definition) do not normally interbreed, nonetheless do produce a hybrid which does not backcross with either parental species, but establishes a new population. There is no change in genome size. The genome of the new species is a 'mixture' or 'mosaic' of the genomes of the two parental species. Homoploid hybridization is also known as recombinational speciation.

Allopolyploidy is the process where two individuals belonging to two different species produce a hybrid individual and this hybrid then undergoes genome doubling and forms a new species. The hybrid individual would (at least in most cases) be infertile without this genome doubling, because the chromosomes of the two parental species are too different to recombine during meiosis. In this case the genome of the new species is the 'sum' of the genomes of the two parental species.

There are different definitions of allopolyploidy. It can be defined in terms of the evolutionary process (first sentence above) or in terms of the behaviour of the chromosomes (along the lines of the second sentence). The definitions usually coincide, but not always.

In the descriptions above, the processes are described in terms of a single individual producing a new species all by itself. That can happen in some species. Plants often reproduce vegetatively, and some flowers can self-fertilize. However the processes may also happen many times, involving many individuals, and possibly over a long time.

Incidence

In general, autopolyploidy, homoploid hybridization, and allopolyploidy are rare in comparison to ordinary speciation. However polyploids (species with doubled or other multiples of genomes compared to close relatives) are common in plants, and also occur in animals and fungi. Around half of flowers and 95% of ferns are polyploids. Polyploids are relatively easy to detect, but it is harder to determine whether their origin was autopolyploidy or allopolyploidy, or to determine how much ordinary speciation has occurred since their origin. The relative incidences of autopolyploidy and allopolyploidy is a matter for current research and debate. Most biologists think that allopolyploidy is much more common. Homoploid hybridization is hard to detect but has been found in a few cases.

Within plants at least, allopolyploidy is very probably the second most important mechanism for generating new species.

References

Most of the information here comes from the book Speciation, by Jerry A. Coyne and H. Allen Orr.

Information about incidence comes from

Tate, Soltis, and Soltis (2005) Polyploidy in plants. In The Evolution of the Genome (T. R. Gregory, ed.)

and

Wood, Takebayashi, Barker, Mayrose, Greenspoon, et al. (2009) The frequency of polyploid speciation in vascular plants. Proc. Natl. Acad. Sci. USA 106.