March 2023 science summary

Happy March!

A paper I'm an author on just came out (James et al. 2023) and I'm super excited about it. It looks at how science and conservation staff at The Nature Conservancy perceive gender equity in their work, and I learned a lot while working on it. Please check out the summary below, and read the whole paper if you have time, or at least read this blog overview.

Also, many of you were intrigued by the use of ChatGPT last month, so I thought I'd try another AI tool to highlight what it does well and what it's limitations are. It's part of elicit.org and it's called "TL;DR papers" (for too long; didn't read). I first read about it here. Elicit does many cool things, but I tested just one feature: uploading a science article and getting: 1) a 1-sentence summary of the abstract, 2) key info about the paper's design and methods, 3) critiques of the paper if it finds them, and 4) the ability to ask specific questions about the paper. I put what I did and the results below under "AI RESEARCH TOOLS." The TL;DR is that I highly recommend the "detailed abstract summary" over the default ones. I also like this guide to things to use AI for (thanks to Bob Lalasz for sending it).

If you know someone who wants to sign up to receive these summaries, they can do so at http://bit.ly/sciencejon (no need to email me).

GENDER AND CONSERVATION:
James et al. 2023 asked over 900 science & conservation staff of The Nature Conservancy about their careers and influence, and how they perceived their gender as impacting that. We found that women had less influence, experienced many barriers to their careers (including harassment, discrimination, and fear of retaliation for speaking out), and that men overestimated gender equity. Only have 5 minutes? Skip to the recommendations on page 7 (we ask orgs to: show public leadership on equity, improve transparency and accountability, diversify teams and improve career pathways for women, be flexible, include training and mentoring as part of broader change, help women connect, address sexual discrimination and harassment, and consider intersectionality). If you have 15 minutes more, read the quotes in Table 2 (p5-8) because they're really compelling and illustrative. Or if you're with the half of men and 3/4 of women in our sample who think we have more to do on gender equity (rather than that we've already "gone overboard" or that it's not an issue as some men reported), just read the whole damn paper because there's a lot of interesting detail and nuance in the results. I learned a ton while helping out on it, and I'm excited to start advocating for the recommendations. You can read it at: https://bit.ly/TNCgenderpaper or a short blog at https://blog.nature.org/science-brief/gender-bias-holds-women-back-in-conservation-careers/ 


AI RESEARCH TOOLS:
I decided to test out Elicit.org on some papers I'm an author on (to make it easy for me to judge how good or bad the info was). I picked my 5 papers that have the most citations (I'm a minor author on all but one of those), the paper which has generated the most response outside of academia (Fisher / Wood / Bradford / Kelsey 2020), and the two most recent papers I've worked on (James et al. 2023 and James et al. 2022). I used the "upload PDFs" option and uploaded all 8 PDFs at once. I pulled info from the summaries, and for each paper asked Elicit one question that I knew the paper had a good answer to. AFTER I was done I discovered Elicit has "detailed summaries" which you get by clicking in "search for paper information", so I went back and added those in with brief comments. I list direct output from Elicit in italics.

Bradford et al. 2019. Soil carbon science for policy and practice. PAPER BLOG
Elicit summary: "Controversy about the role soils might play in climate change mitigation is undermining actions to restore soils for improved agricultural and environmental outcomes."
Jon comments on summary: We had a 2-sentence abstract, and this is the key one. I wish it had the other key message of the paper (we know how to build soil C, and that it can help farmers) but that wasn't in our abstract so that's on us.
Elicit detailed summary: "This paper discusses soil carbon science for policy and practice. It argues that controversy about the role soils might play in climate change mitigation is undermining actions to restore soils for improved agricultural and environmental outcomes." I really like this summary.
Jon comments on other findings: They classified the paper as a literature review, but it's a 'comment' so more opinion although it draws on literature. It flagged we didn't mention our funding which surprised me but it was correct! It was interesting that they said the paper had "0 citations" (it has 88) but then it did list the text from a few papers citing this paper which was cool.
Jon's open question: "What does building soil carbon do?" 
Answer: "Building soil carbon increases yields and reduces erosion." That is pulled from the paper and is accurate.

Fisher et al. 2018. Impact of satellite imagery spatial resolution on land use classification accuracy and modeled water quality. PAPER BLOG
Elicit summary: "The choice of spatial resolution affected the estimated return on investment in watershed conservation for the municipal water company that finances much of this conservation."
Jon comments on summary: I struggled with this one, but have to agree that is the most important sentence from the abstract. I really wish they hadn't cut the 2nd half of the sentence though: ", although it is unlikely that this would have affected the company's decision to invest in the program." My TL;DR would be something like "Using free coarse imagery reduced the accuracy of our water quality modeling, but the coarse data sufficed for the purposes of the intended decision maker (a water treatment company)."
Elicit detailed summary: "This paper compares the impact of satellite image spatial resolution on land use classification and water quality modeling. It finds that higher resolution data is more accurate but also more expensive." This is the key result, but the more interesting implication is how that wouldn't have affected the decision the research was meant to inform.
Jon comments on other findings: The test and outcomes were good, the answer to "who were the participants" was "Brazil" which isn't right. It even successfully pulled the funding source from the paper's main text (this is the only article that worked for). It listed four "possible critiques" of the paper which I was excited to see and evaluate! All four papers basically echoed some of our findings so it was interesting to see they showed up as critiques (they were not): it's a ton of work to use very high resolution data (need to do lots of precise ground references, long analysis times, etc.), data resolution choice can affect your results, and one time when high resolution imagery makes sense is to classify small and heterogeneous features.
Jon's open question: "When is it good to use low resolution data?"
Answer: "The paper concludes by saying that there are key criteria to assist in choosing an appropriate spatial resolution for different contexts, so it depends on the context." This is true, but Table 5 has a summary of what those criteria are, so I was bummed it didn't pull from the list of 12 answers we had to the question like: low budget, large study area, larger patches, need for frequent updates, etc.

Fisher / Wood / Bradford / Kelsey 2020 (we had equal contributions). Improving scientific impact: How to practice science that influences environmental policy and management. PAPER BLOG
Elicit summary: "Scientists who want to improve the impact their research has on decision-making should identify and understand the audience."
Jon comments on summary: Oh boy, this is a case where shorter is not better. We had 4 steps and it cut off the last 3. Here's the sentence it pulled this from "We outline a set of practical steps for scientists who want to improve the impact their research has on decision-making: (a) identify and understand the audience; (b) clarify the need for evidence; (c) gather 'just enough' evidence; and (d) share and discuss the evidence"
Elicit detailed summary: "This paper outlines a set of practical steps for scientists who want to improve the impact their research has on decision-making. These steps include: (a) identifying and understanding the audience; (b) clarifying the need for evidence; (c) gathering "just enough" evidence; and (d) sharing and discussing the evidence." Yup - that's a good summary albeit almost a direct quote!
Jon comments on other findings: N/A (nothing was interesting enough to discuss)
Jon's open question: "Why should scientists read this paper?"
Answer: "The paper provides advice on how to make sure that 1's research has the desired impact." OK, it's weird to say "1's" for "one's", and I was hoping for noting how much research goes to waste, but this is not a bad answer.

James et al. 2022. Gender and conservation science: Men continue to out-publish women at the world’s largest environmental conservation non-profit organization. PAPER BLOG
Elicit summary: "The proportion of women to men remains below gender parity at TNC."
Jon comments on summary: This is a great example of where truncating a sentence makes it inaccurate. While this text does appear in our abstract, it's unclear from Elicit that women are PUBLISHING less scientific articles than men, despite the fact that 45% of conservation and science staff are women.
Elicit detailed summary: "This paper looks at the number of women publishing in conservation science at the Nature Conservancy (TNC) from 1968-2019. It finds that women are underrepresented, making up only 36% of authors, 31% of first authors, and 24% of last authors." This is a good summary and includes probably the most important high level result.
Jon comments on other findings: The question "what did they test" answers "The duration is 1968-2019" (as opposed to something like "The authors tested whether women and men at The Nature Conservancy published science articles at equal rates from 1968-2019"). The outcomes (# women publishing at TNC, proportion of women:men publishing at TNC, proportion of women from the Global South publishing at TNC) are incomplete but true. It didn't find the 3 citations Google know about.
Jon's open question: "How can gender equity in science publishing be improved?"
Answer: "The paper suggests that 'specific strategies and ongoing investments' are needed to address gender diversity among authors, especially women from outside the United States and specifically the Global South." That is frustrating, because it missed that there were seven bulleted and bolded recommendations listed under a heading called "Recommendations for building gender equity in publishing." A better (longer, but actually useful) answer would be "Gender equity in science publishing can be improved via building awareness around bias and systemic barriers, men should collaborate more with women (including in the Global South), include women performing 'invisible labor' as authors, support women to focus on publishing, support women who don't speak English as a 1st language to publish, offer dedicated publishing support for women, and manage impacts of COVID on women" (some of these don't really make sense when shortened like this, though.

James et al. 2023. Gender bias and inequity holds women back in their conservation careers. PAPER
Elicit summary: "Gender bias and inequity holds women back in their conservation careers."
Jon comments on summary: I mean, I'm glad it thinks the title is a good summary! But a summary of the abstract could complement the title. Maybe something like "Surveying about 900 staff of The Nature Conservancy revealed gender equity problems; the authors have recommendations to improve them"?
Elicit detailed summary: "This paper discusses how gender bias and inequity affects women in their conservation careers." I actually think the title is still about as good as this one.
Jon comments on other findings: It got the participants right, and it pulled out a few details relevant to "can I trust this paper?" It couldn't find any citations b/c the paper is so new.
Jon's open question: "What are ways to improve gender equity?"
Answer: "The paper proposes a series of recommendations to improve gender equity in conservation, which involve continual reviewing and adjusting policies, systems, and norms to create a culture that fully leverages the benefits of diversity" Again, I wanted it to know that a question like "what are ways" means I am looking for specifics, not that "ways exist." A better answer could be "The authors recommend that organizations: show public leadership on equity, improve transparency and accountability, diversify teams and improve career pathways for women, be flexible, include training and mentoring as part of broader change, help women connect, address sexual discrimination and harassment, and consider intersectionality."

Kroeger et al. 2014. Reforestation as a novel abatement and compliance measure for ground-level ozone. PAPER BLOG
Elicit summary: "Reforestation is a novel abatement and compliance measure for ground-level ozone."
Jon comments on summary: Again, this is true, but why not go beyond the title? Better to have a longer answer than to waste text by repetition: "Reforestation was predicted to abate ozone at or below the cost of conventional alternatives if land doesn't need to be purchased, and can save money if carbon offsets are generated."
Elicit detailed summary: For some reason Elicit declined to produce a detailed abstract summary for this one, I have no idea why!
Jon comments on other findings: Elicit says "no mention found of funding source" but the paper clearly states that "This analysis was done as part of the collaboration between The Nature Conservancy and Dow Chemical Company and was funded by the Dow Chemical Company Foundation." Also the one paper listed as critique this paper didn't criticize it, and just listed it as a rare case of ecosystem service valuation within industry being published in a peer-reviewed journal.
Jon's open question: "What is the national potential for reforestation to abate ozone?"
Answer: "Reforestation could be a viable, novel approach for abating ground level O3 pollution that complements conventional technology based controls." I was hoping for "Figure 2 maps the places in the contiguous United States where reforestation could be used to abate ozone."

Mcdonald et al. 2009. Urban effects, distance, and protected areas in an urbanizing world. PAPER
Elicit summary: "The median distance from a park to an urban area is already less than 50 km in many regions."
Jon comments on summary: That is not the key point. A better summary would be "A global review of how cities affect protected areas found that 2/3 of effects only apply within 50 km, but this proximity is common in many regions so cities are already having an impact."
Elicit detailed summary: "This paper looks at the effects of urbanization on protected areas. It finds that thousands of protected areas are already impacted by urban areas and many more will be impacted in the future." Not bad, I like my summary better but this is still good.
Jon comments on other findings: Again Elicit missed a partial funding statement "One of us (R.I.M.) was funded by a David H. Smith Conservation Biology Fellowship." It also couldn't access the text of any of this paper's 246 citations!
Jon's open question: "What were the most common threats cities posed to nature?"
Answer: "The most common threats cities posed to nature were "housing and urban areas," "household sewage and urban waste water," "dams & water management," and "logging and wood harvesting."." Sadly this is badly wrong. It misinterpreted a few sentences this was pulled from. To be fair, the best answer is in graphic format (Fig 3) and I don't think this tool can yet read the text of images and extract insights. A better answer would be "Invasive species, housing & urban areas, fire & fire suppression, and dams & water management / use."

Reddy et al. 2017. Advancing Conservation by Understanding and Influencing Human Behavior. PAPER  
Elicit summary: "Integrating two or more behavior change approaches that target multiple, context-dependent factors may be most successful."
Jon comments on summary: True, but a more helpful summary would be broader: "The authors review behavior change models, providing examples and guidance on how to choose an effective one for a given context."
Elicit detailed summary: "This paper discusses how behavioral sciences can help inform conservation policy. It provides a set of guiding questions for applying behavioral insights to conservation policy, including how to define the conservation problem as a behavior change problem, understand behavioral mechanisms, and identify and evaluate behavior change approaches." This is a good summary, no complaints.
Jon comments on other findings: The measured outcomes were wrong, but they got the funding source right (it was in the acknowledgments).
Jon's open question: "What kinds of behavior change interventions use incentives?"
Answer: "The paper mentions that incentives are 1 of 3 behavior change approaches." This is true and it pointed me to the text listing them (awareness, incentives, and nudges). But I was looking for something like "financial or social incentives can be used to create behavior change" as per Fig 3 and a good bit of text. But to be fair I don't think we use the same terms consistently throughout.


REFERENCES:

James, R., Fisher, J. R. B., Carlos-Grotjahn, C., Boylan, M. S., Dembereldash, B., Demissie, M. Z., Diaz De Villegas, C., Gibbs, B., Konia, R., Lyons, K., Possingham, H., Robinson, C. J., Tang, T., & Butt, N. (2023). Gender bias and inequity holds women back in their conservation careers. Frontiers in Environmental Science, 10(January), 1–16. https://doi.org/10.3389/fenvs.2022.1056751 or https://bit.ly/TNCgenderpaper

Sincerely,
 
Jon
 
p.s. This is a photo of our former foster dog Vader letting me know that it was time for hand-chewing and playing, and not time for working.