Skip to main content

You are here

Advertisement

Could More Data Be Less Reliable?

One of the most-cited sources of government data regarding income sources in retirement has revamped its methodology, but in the process led to questions about its reliability of those findings.

A new report from the nonpartisan Employee Benefit Research Institute (EBRI) explains that the Annual Social and Economic Supplement (fielded in March of each year) to the Current Population Survey (CPS) has been shown by other research not only to have misclassified certain types of income, but to have generally under-reported income, specifically sources of retirement income.

That realization led the Census Bureau has redesigned the income questions in the CPS to address the issues from the prior design.

However, as the EBRI report explains, when a questionnaire like this is changed, “the changes can affect results from other areas of the survey that are not changed.” Here the questionnaire was lengthened and new information was given to respondents when asking the additional income-defining questions. Consequently, EBRI notes that more nonresponses could result, and even among those who did respond, “a higher likelihood of incorrect information could be collected due to respondent fatigue and confusion from new questions.”

As for those results, under the traditional survey design, the percentage of all workers found to be working for an employer that sponsored a plan was 50.2%, compared with 47.6% from the redesigned questionnaire, a difference of 2.6 percentage points. For full-time, full-year wage and salary workers ages 21-64 (those most likely to participate in a plan), the difference was even larger at 3.7 percentage points (60.8% traditional versus 57.1% redesigned) and for public-sector workers ages 21-64 the difference was 3.3 percentage points. The percentages of workers participating in an employment-based retirement plan among each workforce were also found to be lower in 2013 under the redesigned questionnaire relative to the traditional questionnaire. Not only were the year-to-year differences from CPS larger from 2013 to 2014 than at any point since at least 1987, the decline contradicted the findings from the Bureau of Labor Statistics’ National Compensation Survey (NCS).

Thus, EBRI notes that while the redesign of the CPS questionnaire achieved one of its primary goals of capturing more income — especially pension income — it appears to have had a serious impact on the results of other variables within the survey: It resulted in sharp declines in the estimated retirement plan participation levels of current workers. Furthermore, those most affected were those groups with the highest likelihoods of participation — those older, with higher earnings, and working for larger employers.

EBRI characterized those decreases in the participation level following the redesign, as well the conflicting time series of participation levels in the CPS relative to other surveys “unexplainable,” raising doubts about the use of CPS participation data, especially for time series trends on retirement participation levels.

Advertisement