Effects of marker type and filtering criteria on QST-FST comparisons

Abstract

Comparative studies of quantitative and neutral genetic differentiation (QST-FST tests) provide means to detect adaptive population differentiation. However, QST-FST tests can be overly liberal if the markers used deflate FST below its expectation, or overly conservative if methodological biases lead to inflated FST estimates. We investigated how marker type and filtering criteria for marker selection influence QST-FST comparisons through their effects on FST using simulations and empirical data on over 18 000 in silico genotyped microsatellites and 3.8 million single-locus polymorphism (SNP) loci from four populations of nine-spined sticklebacks (Pungitius pungitius). Empirical and simulated data revealed that FST decreased with increasing marker variability, and was generally higher with SNPs than with microsatellites. The estimated baseline FST levels were also sensitive to filtering criteria for SNPs: both minor alleles and linkage disequilibrium (LD) pruning influenced FST estimation, as did marker ascertainment. However, in the case of stickleback data used here where QST is high, the choice of marker type, their genomic location, ascertainment and filtering made little difference to outcomes of QST-FST tests. Nevertheless, we recommend that QST-FST tests using microsatellites should discard the most variable loci, and those using SNPs should pay attention to marker ascertainment and properly account for LD before filtering SNPs. This may be especially important when level of quantitative trait differentiation is low and levels of neutral differentiation high.