Recently, I purchased the
Comprehensive Meta-Analysis (CMA) program to help with a particularly tricky meta-analysis that I am
conducting. It involves group
differences that are often calculated through t-test and mixed ANOVAs, and many free meta-analysis programs and scripts cannot interpret these effects. Further yet, many of the studies that I am analyzing do not report all aspects of the effects of interest. In some cases, the authors only report the p-values and nothing else!

So, I decided the purchase the program, CMA, hoping that it would solve all my meta-analysis woes. The program is actually very good, and it can perform most of the analyses that I need; however, there are seven things that I wish I knew about the program before buying it. The first two are good features, while the last five are not-so-good. So, I thought I’d include them on my website for future
buyers.

• CMA can interpret almost any type of statistical input possible. In my meta-analysis alone, I include studies which report two-group pre- and post-test means and standard deviations, two-group post-test means and standard deviations, Cohen’s ds, frequencies, independent group
  p-values, paired-group pre- and post-test means and standard deviations, F-scores, t-values, two-group post-test means and p-values, two-group mean changes and standard deviation of changes, intendent group means and p-values, and paired-group F-scores. As you can tell, the program and handle A LOT of different types of information, and it can automatically transform them into a common effect size. Maybe even more importantly, it can transform these effect sizes into many different other types of effect sizes, such as (in my case) Cohen’s d or Hedge’s g, and it can even
  transform group differences into correlations. This feature of the program is extremely helpful, and was a must-have for my study. Otherwise, I would spend weeks (months?) figuring out the formula for transforming p-values into d-values, and so on.

• CMA provides many sample bias statistics. It can create many different types of funnel plots (cool!). It can also calculate the fail-safe n, and it can apply a trim-and-fill method using a random as well as fixed effects model. In addition to these, I am sure that it can calculate many other sample bias statistics, but these three are the primary ones that I use. Once again, this feature was extremely helpful to me. To be honest, before using the program, I was not knowledgeable about many of the included methods, and I only found out about them after looking through the menus and manuals. Now, I see these analyses everywhere, and many journals seem them as a must-have for meta-analytic results. Good for CMA users!

• CMA cannot correct for unreliability. I was extremely surprised by this
  discovery. Virtually every meta-analysis in my fields (Business/
  I-O Psychology) corrects for unreliability, which makes CMA very difficult to use for many researchers. I even contacted CMA support, and they
  verified that CMA cannot correct for unreliability. The company did ensure that it was a feature that they were trying to develop, but I have no idea when they plan to implement this necessary feature. So, for many people, CMA is not a viable program.

I will say, however, that the CMA output is somewhat friendly with
subsequently applying an artifact distribution method to correct for
unreliability. So, if you know how to apply the artifact distribution method through hand-calculations, it is possible.

• CMA cannot adjust for sample sizes. When correcting for unreliability
  using the individual correction method, it is necessary to adjust for sample sizes. CMA does not do this, making it extremely difficult to apply the
  individual correction method. CMA does, however, provide sample-size-weighted mean effects, which is a benefit. Also, many authors have criticisms against individual correction method and prefer the artifact
  distribution method. So, for many people, this downside of CMA is a non-factor.

• CMA does not provide 80% credibility intervals. This is unfortunate, as most authors report 80% credibility intervals in their articles. I even
  contacted CMA support, and asked whether I was missing the output or if I could calculate the 80% credibility intervals easily with any provided
  output. As of yet, the CMA support has yet to reply to my email. The non-response is unusual, as my other questions were answered within 24 hours. If anyone knows how to easily calculate these intervals, please let me know (email – mch248@psu.edu). Nevertheless, CMA does, however, provide 95% (and other %) confidence intervals.

• CMA does not have a merge function. In my meta-analysis, I have several different categories of outcomes, and I have a separate sheet for each outcome. At the end, I wanted to calculate several overall effects. To do this, I had to manually re-enter the studies’ information. It was a time
  consuming process, and a merge function would have saved ample time. So, when using CMA, it may be helpful to enter all effects into a single sheet. Then, when performing subgroup analyses, save the sheet as a
  different file and remove irrelevant studies. Deleting studies from a sheet is much quicker than manually entering studies into multiple sheets.

• CMA has limited user-created resources. With most statistical programs, I have rarely had to contact the company’s support email address, because other users have made ample guides to solve most problems. No such guides (that I have found) exist with CMA. With that said, CMA has
  released several manuals to help with most problems, and their customer support is fairly speedy; however, I was still left with many questions. This is one of the primary reasons that I created this blog post – so users can know if others have seen their problems. Hopefully, if the program gets more popular in the future, other guides will be created.

Conclusion

Well, those are the seven things that I wish I knew about CMA when I bought it. Overall, it is a really good program. I would consider it a great or amazing
program if it could correct for unreliability (either through individual correction or artifact distribution methods), but I will only consider it a good program until it
includes these methods. If anyone at CMA support reads this post, PLEASE ADD THIS FEATURE!

As always, if you have any questions, please contact me (Matt Howard) at mch248@psu.edu. I am always happy to chat about statistics, research, and many other topics. Thanks for reading!