My Sample is Too Small, Now What?

Specialized Services...

Free Estimate

To request a FREE quote, please CUT AND PASTE the questions below and answer in an e-mail to:

2010@helpwithstatistics.com

Alternates (not hyperlinked):
2010stats @ Gmail.com
Help @ DissertationAdvising.com

***

(01) Your name:

(02) E-mail address:

(03) Day/evening phone numbers (* Required -- in case the response to your e-mail bounces or the editors need clarification regarding the scope of service needed, deadline, etc.):

(04) City, State, Country (or time zone):

(05) Provide a short description of your project and your consulting needs (e.g., data analysis, report writing, charts/graphs, software tutoring):

(06) Are you using a particular brand of statistical software (e.g., SPSS, SAS, JMP, Excel, MatLab)?

(07) What is your academic department / research topic ?

(08) When is your final deadline?

(09) Would you want your consultant to produce tables/graphs/charts?

(10) Would you want your consultant to produce a results narrative?

(11) The name(s) of the statistician(s) you'd like to contact (optional):

(12) How did you learn about our service?:

***

Attach relevant files/documents: (e.g., spreadsheets, university guidelines, SPSS file, proposal, questionnaire, etc.
Please zip large files)

Once your e-mail is received, the network coordinator will forward it (plus any attached files) to the consultant(s) you have selected. If you have not selected consultants, your e-mail will be sent to several consultant(s) chosen by the network coordinator.

If you sent a submission during U.S. business hours and do not get responses within 3 hours, please page the webmaster, and/or resend your submission to the alternate e-mail addresses, and/or leave voicemail for the webmaster: 469-789-3030.

The webmaster cannot quote prices for the freelance statisticians associated with this network. The statistician(s) will contact you directly after receiving your submission and will answer your questions regarding services offered, price, and turnaround time.

Associated Consulting Networks

Thesis and Dissertation Advisors On Call

Technical, Medical, Business, Legal, Education, and Scientific Writing

Thesis and Dissertation Editing

APA Format Experts

Book Editing Associates

Webmaster

Note: The webmaster assures that your submissions receive a response. The webmaster is not a statistician and cannot answer statistics questions. Please submit a request for service to receive a response from a member of our statisticians network.

Have you worked with one (or more) of our consultants? Please send feedback to the network coordinator

My Sample is Too Small, Now What?

by Raymond D. Mooring, Ph.D.

Many social science student researchers use a survey or questionnaire as their primary data gathering instrument. This survey usually includes a number of Likert type scale (1 — Strongly Disagree, 5- Strongly Agree, or some similar variation) questions which the student would like to use to determine a specific group's needs, preferences, desires, satisfaction with a specific item, etc. Unfortunately, many students find themselves in serious trouble when it comes time to analyze their data. So, what are the options if the a particular survey design is inadequate?

To answer this, we must revisit Statistics 101. Many research projects fail in their desired task (usually to gather support for a specific hypothesis) because the researcher did not poll enough respondents. So, as a result, projects with a small sample size suffer from a myriad of limitations. Because sample size directly affects the Type I error and indirectly affects the Type II error and power of a study, small samples usually are not strong enough to provide confidence for its results.

Here's another way to look at this. Statistics is all about studying variations from what is otherwise expected. If one notices large deviations from what is expected, then this is the first indication that the results are significant. That is, the result is not just a result of some random process of chance. The larger the observed difference, the easier it is to find a significant relationship. In other words, the more confident one can be of the existence of an actual relationship. But, here is the cool part: even small differences can be found to be significant if they are based on large samples. In essence, increasing the sample size allows one to detect smaller differences from chance.

This is vitally important, especially in studies where the signal may be slightly different from what is otherwise expected by chance. If this result were found based on a small sample, hypothesis tests (like the t-test, z-test for proportions, and the chi-square test) would tend to not detect the real small signal. Only by increasing repetitions of the observed small difference can one's confidence be built up enough to risk one's reputation in support of the existence of a statistically significant, yet small, real signal.

So, what should the student researcher do if they find themselves in this predicament? First, the student should recognize his problem. The problem is not with the setup of the hypotheses. Rather, the problem is rooted in the study's small sample size. Survey sample sizes less than about 400 will suffer from many limitations. If the student realizes that their sample is small, then he would be best served by acquiring more data. The payoff in the end of the analysis is well worth the additional labor and time required to gather more data. But, if time and resource constraints forbid the researcher from acquiring more data, there are ways to salvage the survey.

If constraints exist, try dichotomizing the variables. In other words, rather than analyze variable with many levels (1- strongly disagree to 5 — strongly agree), consider collapsing levels to only two levels (0 — disagree (collapses 1, 2, 3) and 1 — agree (collapses 4, 5)). This has the effect of increasing the degrees of freedom in the sample, which in turn, allows smaller differences to be found significant. The existence of a relationship can be verified with a chi-squared test. But, unlike the one-sided z-proportion test and the t-test, the researcher does not know the direction of the relationship. In order to determine the direction of the relationship, the phi coefficient needs to be calculated.

Similar to the Pearson correlation coefficient, the phi coefficient determines the strength of the association between nominal variables. It can be calculated easy enough from a contingency table of the two variables. Values close to unity in absolute value indicate strong associations while values between about 0.4 and 0.7 in absolute value indicate moderate associations. The phi coefficient is useful to indicate the associations on all but the most egregiously small data sets (sample sizes smaller than 60 or so). For really small samples, other statistics need to be calculated to determine the association strength.

Finally, the student is advised against calculating descriptive statistics (like means and medians) based on Likert type scale variables. While it is common practice in the social science literature to compare means of Likert scale variables, the author cautions against this type of analysis. Generally speaking, Likert scales provide ordinal data because the data has no set zero (interval data) and fractional values do not have any intuitive sense (like in ratio data). Because of this means and standard deviations are inappropriate. If the researcher wants to calculate means and standard deviations, he should ask the respondents to rate their satisfaction on a scale of 1 to 100 with only the endpoints defined (or similar range). On this scale a rating of 56.5 means something tangible. Moreover, the ratings provide ratio data, which allows for the calculations of descriptive statistics, such as means and standard deviations. Because the numbers in ordinal data are just category labels, descriptive statistics should not be calculated.

In conclusion, when a survey sample size is too small, significant small signals may go undetected. The best way to remedy the situation is to collect more data before analyzing the results. If constraints exist that forbid the researcher from gathering more data than consider dichotomizing the Likert type scale or discarding the ordinal Likert type scale totally in favor of the ratio based rating scale. These procedures will help the researcher in the analysis of the rest of the survey results. If the student can obtain 400 or so samples, then all sorts of analysis techniques will open for him. Moral of the story: get more data! This is how you deal with a small sample size.