[with Sharon F. Piza] (A reply to Fabella’s methodological rejoinder to criticisms of his paper ‘CARP: time to let go’)

If an assertion is made and evidence to support it is shown to be spurious, then that assertion remains pure conjecture. In ”Time to let go of CARP? Not so fast”, we demonstrate how Fabella (2014) misemployed or misconstrued data in the quest to prove that CARP had failed.

To recall, the Fabella paper (2014) made the assertion that as a program to advance the economic welfare of farmers, CARP had accomplished the opposite of its stated goals. To support this assertion, Fabella presented data from some selected studies and  inferred that CARP “failed in its most crucial test” of enhancement of quality of life of beneficiaries thru farm productivity.

We explain in our critique how a more careful and dispassionate reading of the cited studies would not have led a researcher to conclude failure as Fabella did. Fabella did not concoct figures. He just failed to understand their context and limitations. Our critique revealed his “evidence” to be spurious and his arguments refuted. Specifically,

  1. On the observation that productivity in coconut and sugar fell drastically compared to the average in 2011. On the contrary, a comparison of farm yields in agrarian reform communities (ARCs) versus the national average for 2011 (cited by Fabella) and 2005 (overlooked by Fabella) indicate that productivity of major crops under land reform rose more in ARCs. As an example, average yields for coconut and sugar in ARCs gained relative to national averages by 32 and 8 percentage points respectively (see Table 1 of our critique).
  2. On the argument that CARP is redistributing poverty and creating a new class of people which he calls the “landed poor”. On the contrary, the APPC 2007 study, which Fabella lauds as “the most painstaking and careful undertaking to evaluate the performance of CARP on beneficiary welfare”, indicates that poverty incidence in ARC barangays went down by 16 percentage points (from 40% to 24%) between 1990 and 2000, slightly more than in non-ARC barangays.
  3. On the argument that since being an agrarian reform beneificiary (ARB) or residing in an ARC are useless for per capita income advancement, these costly programs are better transformed into direct grants. On the contrary, the APPC 2007 study shows that being an ARB and residing in an ARC have a positive effect on per capita income. [1]   Concretely, the difference in predicted per capita income between ARBs who reside in ARCs and those who do not is approximately 11%; between ARBs who reside in ARCs and land-owning non-ARBs who do not is 8.5%; and between ARBs and land-owning non-ARBs both of which reside in ARCs is 7.5%. [2]  The study also shows that incremental benefits outweigh costs and that the ARC strategy of CARP is far more cost-effective for improving per capita incomes over time than the mainstream agricultural development strategy.

Going back to #2, Fabella’s (2014) paper inappropriately used a poverty estimate from the 2011 ALDA survey (i.e. showing 54% of ARB households are poor) to suggest that CARP was making people poorer. In our critique we already explained why it was inappropriate.[3] Unfortunately, Fabella continues to insist on this ALDA statistic, declaring in his rejoinder that “if there is no better number proffered, the objection is not a falsification: it is just cheap talk!”

Of course, the APPC 2007 study already provides the ‘better’, methodologically defensible numbers and extending its analysis to 2010 would update these numbers (a suggestion Fabella calls ‘more cheap talk’).  Short of an extension study however, we present poverty estimates from the Annual Poverty Indicators Survey (APIS) series, which is able to identify households who acquired agricultural lands through CARP.[4] As Table 1.1 below shows, between 1998 and 2011, there is a greater increase in average per capita income (12.3%) and a deeper reduction in poverty incidence (21%) among CARP households as compared to land-owning non-CARP households and the general population.[5] This seems to reinforce the APPC 2007 findings. Again, an extension of the study to 2010 would allow for a more rigorous analysis.

monsod-tab1

To reiterate, the empirical evidence that Fabella (2014) provides is spurious. His assertion that CARP has failed therefore remains pure conjecture.

In closing we remark on the demand Fabella makes in his rejoinder: that the correct counterfactual for CARP is “a Philippine agriculture with no CARP”. This means, one should imagine what Philippine agriculture would have looked like if the law had never been passed in 1988. Fabella then demands that ”advocates” (which we do not claim to be) must use this comparator to prove that CARP is a success (which we do not claim). We beg to disagree. That counterfactual is ideal but not required of those who seek to measure the impact of CARP on its targeted population. [6]  It is required however, of those who conjecture that there was a CARP-induced credit- or investment-crunch in agriculture since 1988, an economy-wide phenomenon that would nullify and therefore render unmeasurable any achievements of CARP at the beneficiary level. Fabella’s conjecture, Fabella’s obligation to prove.

Footnotes:

[1] See Table 22, p. 74 of APPC 2007. By the way, the reference is APPC 2007 and not  APPC 2008.

[2] This last difference (7.5%) is arguably economically or practically significant even if it may not be statistically significant, which is an important distinction.

[3] See p. 3 par 2. To restate, the ALDA serves a specific internal purpose: to help DAR management monitor and calibrate its plans and programs in ARCs. The meaning, use and comparability of its indicators and estimates are likewise highly specific. In fact, estimates from the household income survey, undertaken for the first time in 2011, are of little use beyond their internal purpose given the survey instrument (e.g. limited definition of ‘income’, recall period of a year) and design (e.g. no comparable sample of non-ARBs in the same ARCs, ARBs who live in comparable barangays that are not ARCs, etc.).   Because of the danger of misuse and misinterpretation by undiscerning researchers, the 2011 household income module of ALDA was not meant for public release. It is just unfortunate that it was featured in Adriano (2013) and picked up by Fabella (2014) without proper disclaimers.

[4] The APIS asks whether a household owns land which it uses for agricultural purposes and, if it does, whether that land was acquired through CARP.

[5] One may ask how poverty incidence can be higher among non-CARP households when average per capita income is also higher. This has to do with the distribution of per capita income among non-CARP households and CARP households.

[6]  In fact, Fabella excused himself from this ideal when he set out to prove that CARP had failed: “It is admittedly difficult to estimate even just the net global economic welfare benefit of a government intervention, although some such have been attempted for land reform elsewhere … It is much simpler to concentrate on just the welfare improvement of the target population.” (Fabella 2014, p.2 par 1).  Methodologically, it seeks to answer the question ‘what has been the impact of CARP on Philippine agriculture and the economy?’ which is arguably different from ‘what has been the impact of CARP on its targeted beneficiaries?’ The second question can be explored using a less ambitious counterfactual, as students of impact evaluation will know, and findings are not invalidated if the ideal cannot be implemented.