The endeavor for greater food security has caused trade-offs between increasing agricultural production and conserving habitat of threatened species. We take a novel approach to analyze these trade-offs by applying and comparing three systems methods (systems diagrams, influence matrices, and land use modeling) in a case study of Uganda. The first two methods were used to scope out the trade-off system and identify the most important variables influencing trade-offs. These variables were agricultural yield, land governance processes, and change in land use and land cover. The third method was used to quantify trade-offs and evaluate policy scenarios to alleviate them. A reference scenario indicated that increasing agricultural production by expanding agricultural land provided food for 79% more people in 2050 (compared to 2005) but with a 48% loss of habitat of threatened species. A scenario assuming strong investments to augment agricultural yield increased the number of people fed in 2050 up to 157%, while reducing the loss of habitat down to 27%. We use a novel “trade-off coefficient” for a consistent comparison of scenario results. A scenario assuming yield improvement and ring-fencing protected areas reduced the trade-off coefficient from − 0.62 in the reference case to − 0.15. This coefficient can be used as a common basis to compare results from different trade-off studies. It was found that the three systems methods are useful, but have limitations as stand-alone tools. Combining the methods into a single methodology increases their collective utility by maximizing the transparency and comprehensiveness and potential stakeholder engagement of a trade-off analysis.