Don’t give it away: the hidden risks of generative AI for scientistsThis text aims to provide readers with an overview of the artificial intelligence (AI) that powers conversational agents and has become commonplace in recent years, after the launch of ChatGPT (https://chatgpt.com). Also, we outline some benefits and potential risks associated with the misuse of this technology in the scientific research ecosystem. Finally, we suggest ways to mitigate these risks.
AI is a field of computer science that emerged in the 1950s. The term was coined by McCarthy et al. (2006) in the proposal for the Dartmouth Summer Research Project on Artificial Intelligence to organize a workshop during the summer of 1956 at Dartmouth College in Hanover, United States. The workshop aimed to discuss questions related to what McCarthy et al. (2006) called AI issues. That summer, discussions covered, among other topics, how computers could use natural language from a word-manipulation perspective and what the formal modeling of a neural network would look like. Since then, AI has gone through phases of euphoria and despair, mostly due to the processing limitations of computers.
Currently, in addition to the exponential increase in processing power, new AI techniques for text analysis have made it possible for computers to use natural language effectively. This was one of the questions raised in the summer of 1956 and is properly answered now. This advancement in AI has enabled the penetration of intelligent services throughout society, increasing people’s perception of how much this technology can benefit them. This perception, combined with the integration of intelligence into several services provided to society – such as mobile communication services and social networks – causes the public to quickly adopt technology without understanding that there are also risks associated with its use.
Naturally, the ecosystem of scientific research can benefit from its use. Nevertheless, additional care is required to prevent AI misuse from affecting its processes, leading to unpredictable and damaging results. To better understand the risks, it is important to identify which intelligence we are referring to.
Given the very characteristics of AI, there is no single definition for the term. Russell; Norvig (2021) adopt the idea of an “agent” to define the intelligence attributed to computational systems. In this case, an agent is an entity that possesses sensors and actuators and is immersed in an environment. Its intelligence lies in how this entity processes the information collected from the environment by its sensors and in the actions resulting from this processing, which can modify the environment through the actuators. Thus, the authors organize AI definitions into four dimensions:
-
Intelligence underlying systems that act like humans;
-
Intelligence underlying systems that reason like humans;
-
Intelligence underlying systems that reason rationally;
-
Intelligence underlying systems that act rationally.
When we talk about an intelligence that reasons and acts rationally, we refer to an intelligence that allows the agent to use logic or rules to think and justify its actions in deterministic situations or, in the presence of uncertainty, through probabilistic models. In this case, to analyze the reasons that led the agent to take certain decisions is explainable.
On the other hand, when the intelligence in question is one that acts like a human being, we refer to an intelligence capable of communicating in natural language, storing acquired knowledge, reasoning, learning, and inferring new things using such knowledge, perceiving the world, and manipulating its objects. To think like humans, this intelligence needs to acquire cognitive capacity that mimics human ability. In this sense, several areas of study have been dedicated to researching how to perform each of these tasks like humans, namely: natural language processing, knowledge representation, automated reasoning, machine learning, computer vision, and robotics.
Despite this separation into areas of study, machine learning is a field that permeates the others, particularly computer vision, natural language processing, and automated reasoning. In fact, since the early 2000s, we have witnessed an unprecedented evolution of machine learning techniques, driven by increased processing capacity and the vast availability of data to feed the learning process. These techniques were primarily dedicated to solving tasks related to image and text processing. Essentially, underlying these techniques are pattern recognition and the probabilistic inference of the next pattern, without any semantic understanding of the processed content. Furthermore, patterns are learned from data collected in the past, which may be recent, but is still the past.
Beginning in mid-2017, with the publication of the article “Attention is all you need” (referring here to the 2023 revision) by a group of Google researchers (Vaswani et al., 2017), a new way of processing text was presented, making text translation tasks faster and more accurate. From then on, intelligent solutions based on this form of processing were created for other tasks associated with natural language text processing. For this evolution to occur, massive amounts of text had to be used to train these solutions, leading to the creation of large language models (LLMs). These models are the foundation of intelligent services currently available to the general public, such as ChatGPT, Claude (https://claude.ai/), Gemini (https://gemini.google.com), and Perplexity (https://www.perplexity.ai/). An important note is regarding the energy and environmental costs associated with these services. Training these LLMs and maintaining the services associated with their use requires significant processing, leading to high consumption of energy and water (needed to cool the machines). But that could be the subject of another essay.
Returning to intelligent services, they offer users an interface that receives general questions in natural language and returns answers in natural language that are grammatically correct, contextually consistent, but not necessarily accurate. This response is generated based on the patterns learned by the model and the context in which it is placed. The underlying AI of this type of service generates responses that can repeat existing patterns, present them in combined ways, or infer new text content, which in turn can be transformed into various formats, such as image, audio, or video. This AI is called generative AI, a type of machine learning that uses a technique called deep neural networks to simulate the learning and decision-making processes of the human brain. This is why it is possible to dialogue with agents like ChatGPT and feel as though the answers could have been crafted by human beings. And that is where the risks arise, and why I continue to advocate for the need to train people for the (proper) use of AI.
Since knowledge evolves based on existing knowledge, LLMs have evolved from the knowledge generated through human interactions with systems like ChatGPT by prompting questions or requests. The business model of big techs consists of providing their services “for free” in exchange for using all the content of the prompts and their associated results. This means that while prompting with ChatGPT in a free account, all the knowledge from the conversation will be used to improve the LLM that feeds the knowledge of ChatGPT.
Considering the scientific research ecosystem, the adoption of AI services to support its methods and processes is a reality that must be continuously monitored, considering scientific integrity. In this ecosystem, a fundamental part is the writing and dissemination of research results, since new knowledge is highly dependent on existing knowledge. Understanding how generative AI works and Big Tech’s business model reveals that some phases of the scientific writing and publishing process may provide content to improve these LLMs even before they are published in appropriate venues. This occurs whenever a free service is used, for example, to improve a text to fit a journal’s style or even to translate it into another language. Using these services with an institutional account can prevent this, as such service agreements usually include clauses of a data process agreement. Such clauses ensure privacy and property of all customer data, including prompts and associated answers, in addition to compliance with data regulations. Nevertheless, AI can improve research results by supporting the literature review, text summarization, and hypothesis refinement, among others.
This text is an alarm to avoid using AI services to support scientific research without due diligence. In fact, all free-of-charge services have a price, and it is exactly all the data you prompt on them. Be aware: do not give for free something that you spent a lifetime creating before having the credits for it. Don’t give it away!
ACKNOWLEDGEMENTS
The author is an associate researcher at the Center for Artificial Intelligence (C4AI) and acknowledges the financial support provided by FAPESP (grant #2019/07665-4).
-
DECLARATION OF USE OF ARTIFICIAL INTELLIGENCE TOOLS
The author uses Overleaf with its embedded AI as the text editor. This use helps to avoid typos and grammar errors.
-
FUNDING
Not applicable.
-
ETHICAL APPROVAL
Not applicable.
AVAILABILITY OF DATA AND MATERIAL
Not applicable.
REFERENCES
-
MCCARTHY, J; MINSKY, M. L.; ROCHESTER, N.; SHANNON, C. E. A proposal for the Dartmouth summer research project on artificial intelligence, August 31, 1955. AI Magazine, v. 27, n. 4, p. 12-14, 2006. https://doi.org/10.1609/aimag.v27i4.1904
» https://doi.org/10.1609/aimag.v27i4.1904 - RUSSELL, Stuart; NORVIG, Peter. Artificial intelligence: a modern approach Global ed. Harlow: Pearson Education, 2021.
-
VASWANI, A.; SHAZEER, N.; PARMAR, N.; USZKOREIT, J.; JONES, Ll.; GOMEZ, A. N.; KAISER, L.; POLOSUKHIN, I. Attention is all you need arXiv, 2017. Available at: https://arxiv.org/abs/1706.03762 Access on: 20 feb. 2026.
» https://arxiv.org/abs/1706.03762
Edited by
-
Associate Editor:
Silvia Galleti https://orcid.org/0000-0002-0745-5716
