Trust evaluation in virtual software development teams using BERT-based language models

Sergio Zapata; Facundo Gallardo; Gustavo Sevilla; Estela Torres; Raymundo Forradellas

doi:10.24215/16666038.23.e04

Authors

Sergio Zapata Instituto de Informática - Universidad Nacional de San Juan https://orcid.org/0000-0002-8129-9395
Facundo Gallardo Informatic Institute, Universidad Nacional de San Juan, Argentina https://orcid.org/0000-0001-9057-3790
Gustavo Sevilla Informatic Institute, Universidad Nacional de San Juan, Argentina https://orcid.org/0000-0001-9070-0948
Estela Torres Informatic Departament, Universidad Nacional de San Juan, Argentina https://orcid.org/0000-0002-8845-9777
Raymundo Forradellas Intelligent Systems Laboratory, Universidad Nacional de Cuyo, Argentina https://orcid.org/0000-0001-6049-2439

DOI:

https://doi.org/10.24215/16666038.23.e04

Keywords:

BERT-based language model, Social software engineering, Trust analysis

Abstract

Nowadays, people from different geographical areas can be closely related thanks to advances in information and communication technologies. This has a greater impact in software development organizations where their members form virtual work teams. In these new co-located work scenarios, the construction of interpersonal trust is more complex and its impact is very relevant in the performance of software development teams. This paper presents the results of the performance
evaluation of four pre-trained language models based on BERT applied to trust analysis tasks. For this work, a small dataset of 1453 comments obtained from software projects stored on Github was created. The evaluated language models achieved moderately good values, in the order of 0.84 for the F1-score metric, which augurs that with further research they could be significantly improved.

Downloads

Download data is not yet available.

References

I. Aza. “Man as Subject of Internet Communication”. In International Conference Communicative Strategies of Information Society (CSIS 2018) (pp. 383-388). Atlantis Press. 2018.

B. H. Malik, S. Faroom, M. N. Ali, N. Shehzad, S. Yousaf, H. Saleem, K. Khan. “Geographical Distance and Communication Challenges in Global Software Development: A Review”. International Journal of Advance Computer Science and Applications. Vol. 9, N° 5, pp. 406-414. 2018.

M. Alsharo, D. Gregg, R. Ramirez. “Virtual team effectiveness: The role of knowledge sharing and trust”. Information & Management. Vol. 54 N° 4, pp. 479-490. 2017.

S. Morrison-Smith, J. Ruiz. “Challenges and barriers in virtual teams: a literature review”. SN Applied Sciences, 2(6), 1-33. 2020.

L. F. Capretz, F. Ahmed. “Making sense of software development and personality types”. IT professional. Vol. 12, N° 1, pp. 6-13.2010.

A. Cockburn, J. Highsmith. “Agile software development, the people factor”. Computer. Vol. N° 11, pp. 131-133. 2001.

D. Graziotin, X. Wang, P. Abrahamsson. “Happy software developers solve problems better: psychological measurements in empirical software engineering”. Peer J. Vol. 2, e289. 2014.

N. Novielli, F. Calefato, F. Lanubile. “Towards discovering the role of emotions in stack overflow”. Proceedings of the 6th international workshop on social software engineering ACM. pp. 32-36. 2014.

M. Ortu, G. Destefanis, S. Counsell, S. Swift, R. Tonell, M. Marchesi. “How diverse is your team? Investigating gender and nationality diversity in GitHub teams”. Journal of Software Engineering Research and Development. Vol. 5 N°1, pp. 1-18. 2017.

M. Hertzum. “The importance of trust in software engineers' assessment and choice of information sources”. Information and Organization. Vol. 12, N° 1, pp 1-18. 2002.

N. B. Moe, D. Šmite. “Understanding a lack of trust in Global Software Teams: a multiple-case study”. Software Process: Improvement and Practice. Vol. 13, N° 3, pp. 217-231. 2008.

M. Grabowski, K. H. Roberts. “Risk mitigation in virtual organizations”. Organization Science. Vol. 10, N° 6, pp. 704-721. 1999.

P. Kanawattanachai, Y. Yoo. “Dynamic nature of trust in virtual teams”. The Journal of Strategic Information Systems. Vol. 11, N° 3-4, pp. 187-213. 2002.

L. L. Martins, L. L.Gilson, M. T. Maynard. “Virtual teams: What do we know and where do we go from here?”. Journal of management. Vol. 30, N° 6, pp. 805-835. 2004.

S. L. Jarvenpaa, T. R. Shaw, D. S. Staples. “Toward contextualized theories of trust: The role of trust in global virtual teams”. Information systems research. Vol. 15, N° 3, pp. 250-267. 2004.

R. C. Mayer, J. H. Davis, F. D. Schoorman. “An integrative model of organizational trust”. Academy of management review. Vol. 20, N° 3, pp. 709-734. 1995.

J. D. Lewis, A. Weigert. “Trust as a social reality”. Social Forces. Vol 63, N°4, pp. 967-985. 1985.

Clark W. R., Clark L. A. and Crossley K.: “Developing multidimensional trust without touch in virtual teams”. Marketing Management Journal, 20(1) (2010) 177-193.

J. Wilson, S. Straus, B. McEvily. “All in due time: The development of trust in computer mediated and face-to-face teams”. Organizational Behavior and Human Decision Processes. Vol. 99, N° 1, pp. 16-33. 2006.

D. J. McAllister. “Affect and cognition-based trust as foundations for interpersonal cooperation in organisations”. Academy of Management J. Vol. 38, N°1, pp. 25-59. 1995.

M. A, Rothenberger, Y. C. Kao, L. N. Van Wassenhove. ”Total quality in software development: An empirical study of quality drivers and benefits in Indian software projects”. Information & Management. Vol. 47, N° 7-8, pp. 372-379. 2010.

A. Gopal, M. S. Krishnan, T. Mukhopadhyay, D. R. Goldenson. “Measurement programs in software development: determinants of success”. IEEE Transactions on software engineering. Vol. 28, N°9, pp. 863-875. 2002.

F. Q. B. da Silva, R. Prikladnicki, A. C. C. França, C. V. F. Monteiro, C. Costa, R. Rocha. “An evidencebased model of distributed software development project management: results from a systematic mapping study”. Journal of Software: Evolution and Process. Vol 24, N° 6, pp. 625–642. 2012.

N. B. Moe, D. S. Cruzes, T. Dybå, E. Engebretsen. “Coaching a Global Agile Virtual Team”. Proceedings of the 2015 IEEE 10th International Conference on Global Software Engineering. Vol I, Washington, DC, USA, pp. 33-37. 2015.

D. Thomas, R. Bostrom. “Building Trust and Cooperation Through Technology Adaptation in Virtual Teams: Empirical Field Evidence”. Information Systems Management. Vol. 25, N °1, pp. 45-56. 2010.

A. L. McNab, K. A. Basoglu, S. Sarker, Y. Yu. “Evolution of cognitive trust in distributed software development teams: A punctuated equilibrium model”. Electronic Markets. Vol. 22, N°1, pp. 21-36. 2012.

F. Calefato, D. Gendarmi, F. Lanubile. “Embedding Social Networking Information into Jazz to Foster Group Awareness within Distributed Teams”. Proceedings of the 2nd Int Workshop on Social Software Engineering and Applications. Vol I, pp. 23-28. 2009.

V. Casey. “Developing trust in virtual software development teams”. Journal of theoretical and applied electronic commerce research. Vol.5 (2) (2010) 41-58.

R. Ocker. “Enhancing Learning Experiences in Partially Distributed Teams: Training Students to Work Effectively Across Distances”. Proceedings 42nd Hawaii International Conference on System Sciences. Vol. I, Washington, DC, USA, pp. 1–10. 2009.

L. Tan, A. Hindle. “Guest Editorial: Special Section on Mining Software Repositories”. Empirical Software Engineering. Vol. 24, pp. 1-3. 2019.

Nugroho, K. S., Sukmadewa, A. Y., & Yudistira, N. (2021, September). “Large-scale news classification using bert language model: Spark nlp approach”. In 6th International Conference on Sustainable Information Engineering and Technology 2021 (pp. 240-246).

Dai, J., & Chen, C. (2020, August). “Text classification system of academic papers based on hybrid Bert-BiGRU model”. In 2020 12th International Conference on Intelligent Human- Machine Systems and Cybernetics (IHMSC) (Vol. 2, pp. 40-44). IEEE.

Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). “Bert: Pre-training of deep bidirectional transformers for language understanding”. arXiv preprint arXiv:1810.04805.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). “Roberta: A robustly optimized bert pretraining approach”. arXiv preprint arXiv:1907.11692.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). “Language models are unsupervised multitask learners”. OpenAI blog, 1(8), 9.

Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V. (2019). “Xlnet: Generalized autoregressive pretraining for language understanding”. Advances in neural information processing systems, 32.

Arase, Y., & Tsujii, J. (2021). “Transfer fine-tuning of BERT with phrasal paraphrases”. Computer Speech & Language, 66, 101164.

Song, H., Wang, Y., Zhang, K., Zhang, W. N., & Liu, T. (2021). “BoB: BERT over BERT for training persona-based dialogue models from limited personalized data”. arXiv preprint arXiv:2106.06169.

González-Carvajal, S., & Garrido-Merchán, E. C. (2020). “Comparing BERT against traditional machine learning text classification”. arXiv preprint arXiv:2005.13012.

M. Niazi, N. Ikram, M. Bano, S. Imtiaz, S. U. Khan. “Establishing trust in offshore software outsourcing relationships: an exploratory study using a systematic literature review”. IET software. Vol. 7, N° 5, pp. 283-293. 2013.

Zapata, S., Barros-Justo, J. L., Matturro, G., & Sepúlveda, S. (2021). “Measurement of interpersonal trust in virtual software teams: A systematic literature review”. INGENIARE-Revista Chilena de Ingeniería, 29(4).

F. Q. B. da Silva, R. Prikladnicki, A. C. C. França, C. V. F. Monteiro, C. Costa, R. Rocha. “An evidencebased model of distributed software development project management: results from a systematic mapping study”. Journal of Software: Evolution and Process. Vol 24, N° 6, pp. 625–642. 2012.

S. Tyagi, R. Sibal, B. Suri. “Role of trust in distributed agile software development teams- A light weight systematic literature review”. ICTACT Journal on Management Studies. Vol. 4, N° 2, pp. 748-753. 2018.

Uddin, G., Guéhénuc, Y. G., Khomh, F., & Roy, C. K. (2022). “An Empirical Study of the Effectiveness of an Ensemble of Stand-alone Sentiment Detection Tools for Software Engineering Datasets”. ACM Transactions on Software Engineering and Methodology (TOSEM), 31(3), 1-38.

Obaidi, M., Nagel, L., Specht, A., & Klünder, J. (2022). “Sentiment analysis tools in software engineering: A systematic mapping study”. Information and Software Technology, 107018.

Paul, S., & He, F. (2012, January). “Time pressure, cultural diversity, psychological factors, and information sharing in short duration virtual teams”. In 2012 45th Hawaii International Conference on System Sciences (pp. 149-158). IEEE

A. Mitchell, I. Zigurs, Trust in virtual teams: solved or still a mystery? ACM SIGMIS Database 40 (3) (2009) 61–83, http://dx.doi.org/10.1145/1592401.1592407.

Jarvenpaa, S. L., Knoll, K., & Leidner, D. E. (1998). “Is anybody out there? Antecedents of trust in global virtual teams”. Journal of management information systems, 14(4), 29-64.

] F.-y. Kuo, C.-p. Yu, An exploratory study of trust dynamics in work-oriented virtual teams, J. Comput.- Mediated Commun. 14 (4) (2009) 823–854, http://dx.doi.org/10.1111/j.1083-6101.2009.01472.x.

Khan, M. S. (2012). Role of trust and relationships in geographically distributed teams: exploratory study on development sector. International Journal of Networking and Virtual Organisations, 10(1), 40-58.

Canete, J., Chaperon, G., Fuentes, R., Ho, J. H., Kang, H., & Pérez, J. (2020). “Spanish pre-trained bert model and evaluation data”. Pml4dc at iclr, 2020, 1-10.

Tiedemann, J. (2012, May). “Parallel data, tools and interfaces in OPUS”. In Lrec (Vol. 2012, pp. 2214- 2218).

Carrino, C. P., Costa-jussà, M. R., & Fonollosa, J. A.

(2019). “Automatic spanish translation of the squad dataset for multilingual question answering”. arXiv preprint arXiv:1912.05200.

Van Rijsbergen, C. (1979, September). “Information retrieval: theory and practice”. In Proceedings of the Joint IBM/University of Newcastle upon Tyne Seminar on Data Base Systems (Vol. 79).

Sun, C., Qiu, X., Xu, Y., & Huang, X. (2019, October). “How to fine-tune bert for text classification?”. In China national conference on Chinese computational linguistics (pp. 194-206). Springer, Cham.

Moon, A., & Raju, T. (2013). “A survey on document clustering with similarity measures”. International Journal of Advanced Research in Computer Science and Software Engineering, 3(11), 599-601.

Wright, R. E. (1995). “Logistic regression”. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding multivariate statistics (pp. 217–244). American Psychological Association.

Radford, A., Wu, J., Amodei, D., Amodei, D., Clark, J., Brundage, M., & Sutskever, I. (2019). Better language models and their implications. OpenAI blog, 1, 2.