Ontology based Feature Selection and Weighting for Text classification using Machine Learning
Keywords:
Text Classification, Feature selection, Feature weighting, Machine Learning (ML), Ontology, WordNetAbstract
Text classification consists in attributing text (document) to its corresponding class (category). It can be performed using an artificial intelligence technique called machine learning. However, before training the machine learning model that classifies texts, three main steps are also mandatory: (1) Preprocessing, which cleans the text; (2) Feature selection, which chooses the features that significantly represent the text; and (3) Feature weighting, which aims at numerically representing text through feature vector. In this paper, we propose two algorithms for feature selection and feature weighting. Unlike most existing works, our algorithms are sense-based since they use ontology to represent, not the syntax, but the sense of a text as a feature vector. Experiments show that our approach gives encouraging results compared to existing works. However, some additional suggested improvements can make these results more impressive.
Downloads
References
K. Nalini and L. J. Sheela, "Survey on text classification," International Journal of Innovative Research in Advanced Engineering, vol. 1, pp. 412-417, 2014.
W. B. Croft, D. Metzler, and T. Strohman, Search engines: Information retrieval in practice vol. 520: Addison-Wesley Reading, 2010.
C. Lanquillon, "Enhancing text classification to improve information filtering," Otto-von-Guericke-Universität Magdeburg, Universitätsbibliothek, 2001.
R. Du, R. Safavi-Naini, and W. Susilo, "Web filtering using text classification," in The 11th IEEE International Conference on Networks, 2003. ICON2003., Sydney, NSW, Australia, 2003, pp. 325-330.
A. Bhowmick and S. M. Hazarika, "E-Mail Spam Filtering: A Review of Techniques and Trends," in Advances in Electronics, Communication and Computing, A. Kalam, S. Das, and K. Sharma, Eds., ed Singapore: Springer Singapore, 2018, pp. 583-590.
K. Lang, "NewsWeeder: Learning to Filter Netnews," in Machine Learning Proceedings 1995, A. Prieditis and S. Russell, Eds., ed San Francisco (CA): Morgan Kaufmann, 1995, pp. 331-339.
B. Liu and L. Zhang, "A Survey of Opinion Mining and Sentiment Analysis," in Mining Text Data, C. C. Aggarwal and C. Zhai, Eds., ed Boston, MA: Springer US, 2012, pp. 415-463.
M. Heidarysafa, K. Kowsari, L. Barnes, and D. Brown, "Analysis of Railway Accidents' Narratives Using Deep Learning," in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, Florida, USA, 2018, pp. 1446-1453.
I. Mani and M. T. Maybury, Advances in Automatic Text Summarization, abridged, illustrated, reprint ed.: MIT Press, 1999.
A. L. Samuel, "Some Studies in Machine Learning Using the Game of Checkers," IBM Journal of Research and Development, vol. 3, pp. 210-229, 1959.
G. A. Miller, "WordNet: a lexical database for English," Commun. ACM, vol. 38, pp. 39–41, 1995.
X. Zhou, R. Gururajan, Y. Li, R. Venkataraman, X. Tao, G. Bargshady, P. D. Barua, and S. Kondalsamy-Chennakesavan, "A survey on text classification and its applications," Web Intelligence, vol. 18, pp. 205-216, 2020.
J. Chen, H. Huang, S. Tian, and Y. Qu, "Feature selection for text classification with Naïve Bayes," Expert Systems with Applications, vol. 36, pp. 5432-5435, 2009/04/01/ 2009.
S. Kullback and R. A. Leibler, "On Information and Sufficiency," The Annals of Mathematical Statistics, vol. 22, pp. 79-86, 1951.
J. R. Quinlan, "Induction of decision trees," Machine Learning, vol. 1, pp. 81-106, 1986/03/01 1986.
K. Pearson, "X. On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling," The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, vol. 50, pp. 157-175, 1900/07/01 1900.
Y. Zhai, W. Song, X. Liu, L. Liu, and X. Zhao, "A Chi-Square Statistics Based Feature Selection Method in Text Classification," in 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, 2018, pp. 160-163.
L. Galavotti, F. Sebastiani, and M. Simi, "Experiments on the Use of Feature Selection and Negative Evidence in Automated Text Categorization," in Proceedings of the 4th European Conference on Research and Advanced Technology for Digital Libraries, Lisbon, Portugal, 2000, pp. 59-68.
H. T. Ng, W. B. Goh, and K. L. Low, "Feature selection, perceptron learning, and a usability case study for text categorization," in Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval, Philadelphia, Pennsylvania, USA, 1997, pp. 67–73.
C. E. Shannon, "A mathematical theory of communication," The Bell System Technical Journal, vol. 27, pp. 379-423, 1948.
R. M. Fano, "Transmission of Information: A Statistical Theory of Communications," American Journal of Physics, vol. 29, pp. 793-794, 1961.
D. Agnihotri, K. Verma, and P. Tripathi, "Mutual information using sample variance for text feature selection," in Proceedings of the 3rd International Conference on Communication and Information Processing, Tokyo, Japan, 2017, pp. 39–44.
W. J. Wilbur and K. Sirotkin, "The automatic identification of stop words," Journal of information science, vol. 18, pp. 45-55, 1992.
Y. Yang, "Noise reduction in a statistical approach to text categorization," in Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, Seattle, Washington, USA, 1995, pp. 256–263.
Y. Yang and J. Wilbur, "Using corpus statistics to remove redundant words in text categorization," Journal of the American Society for Information Science, vol. 47, pp. 357-369, 1996.
M. Szumilas, "Explaining odds ratios," Journal of the Canadian academy of child and adolescent psychiatry, vol. 19, pp. 227-229, 2010.
D. Mladenić, "Feature subset selection in text-learning," in Machine Learning: ECML-98, Berlin, Heidelberg, 1998, pp. 95-100.
W. Shang, H. Huang, H. Zhu, Y. Lin, Y. Qu, and Z. Wang, "A novel feature selection algorithm for text categorization," Expert Systems with Applications, vol. 33, pp. 1-5, 2007/07/01/ 2007.
D. Shen, J.-T. Sun, Q. Yang, H. Zhao, and Z. Chen, "Text Classification Improved through Automatically Extracted Sequences," in Proceedings of the 22nd International Conference on Data Engineering (ICDE'06), Atlanta, GA, USA, 2006, pp. 121-121.
T. R. Gruber, "A translation approach to portable ontology specifications," Knowledge Acquisition, vol. 5, pp. 199-220, 1993/06/01/ 1993.
R. Stevens, C. A. Goble, and S. Bechhofer, "Ontology-based knowledge representation for bioinformatics," Briefings in Bioinformatics, vol. 1, pp. 398-414, 2000.
A. Kehagias, V. Petridis, V. G. Kaburlasos, and P. Fragkou, "A Comparison of Word- and Sense-Based Text Categorization Using Several Classification Algorithms," Journal of Intelligent Information Systems, vol. 21, pp. 227-247, 2003/11/01 2003.
A. Moschitti, "Syntactic and semantic kernels for short text pair categorization," in Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, Athens, Greece, 2009, pp. 576–584.
X. Peng and B. Choi, "Document Classifications based on Word Semantic Hierarchies," in Proceedings of the International Conference on Artificial Intelligence and Applications (AIA’05), Innsbruck, Austria, 2005, pp. 362-367.
Z. S. Harris, "Distributional Structure," WORD, vol. 10, pp. 146-162, 1954/08/01 1954.
M. F. McTear, Z. Callejas, and D. Griol, The conversational interface, 1 ed. vol. 6: Springer Cham, 2016.
K. Kowsari, K. Jafari Meimandi, M. Heidarysafa, S. Mendu, L. Barnes, and D. Brown, "Text Classification Algorithms: A Survey," Information, vol. 10, p. 150, 2019.
D. Jurafsky and J. H. Martin, Speech and Language Processing, Third Edition draft ed., 2021.
A. Rajaraman and J. D. Ullman, Mining of Massive Datasets: Cambridge University Press, 2011.
H. P. Luhn, "A Statistical Approach to Mechanized Encoding and Searching of Literary Information," IBM Journal of Research and Development, vol. 1, pp. 309-317, 1957.
K. Sparck Jones, "A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL," Journal of Documentation, vol. 28, pp. 11-21, 1972.
T. Mikolov, K. Chen, G. Corrado, and J. Dean. (2013, October 2022). Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781. Available: https://ui.adsabs.harvard.edu/abs/2013arXiv1301.3781M
T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, "Distributed representations of words and phrases and their compositionality," Advances in neural information processing systems, vol. 26, 2013.
Q. Le and T. Mikolov, "Distributed Representations of Sentences and Documents," in Proceedings of the 31st International Conference on Machine Learning, Beijing China, 2014, pp. 1188--1196.
L. Buitinck, G. Louppe, M. Blondel, F. Pedregosa, A. Mueller, O. Grisel, V. Niculae, P. Prettenhofer, A. Gramfort, J. Grobler, R. Layton, J. Vanderplas, A. Joly, B. Holt, and G. Varoquaux. (2013, October 2022). API design for machine learning software: experiences from the scikit-learn project. arXiv:1309.0238. Available: https://ui.adsabs.harvard.edu/abs/2013arXiv1309.0238B
K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J. Attenberg, "Feature hashing for large scale multitask learning," in Proceedings of the 26th Annual International Conference on Machine Learning, Montreal, Quebec, Canada, 2009, pp. 1113–1120.
G. Grefenstette, "Tokenization," in Syntactic Wordclass Tagging, H. van Halteren, Ed., ed Dordrecht: Springer Netherlands, 1999, pp. 117-133.
M. Toman, R. Tesar, and K. Jezek, "Influence of word normalization on text classification," in Proceeding of Multidisciplinary Approaches to Global Information Systems, InSciT 2006, Merida, Spain, 2006, pp. 354-358.
S. Bird, E. Klein, and E. Loper, Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit: O'Reilly Media, 2009.
V. Vapnik and A. Chervonenkis, "A note on one class of perceptrons," Automation and Remote Control, vol. 25, pp. 821-837, 1964.
V. Vapnik and A. Lerner, "Pattern recognition using generalized portrait method," Automation and Remote Control, vol. 24, pp. 774-780, 1963.
C. Cortes and V. Vapnik, "Support-vector networks," Machine Learning, vol. 20, pp. 273-297, 1995/09/01 1995.
D. Greene and P. Cunningham, "Practical solutions to the problem of diagonal dominance in kernel document clustering," in Proceedings of the 23rd international conference on Machine learning, Pittsburgh, Pennsylvania, USA, 2006, pp. 377–384.
P. S. Foundation. (2022, October 2022). Python 3.10.7 documentation. Available: https://docs.python.org/3/
S. Loria. (2020, October 2022). textblob Documentation. Release 0.16.0. Available: https://buildmedia.readthedocs.org/media/pdf/textblob/latest/textblob.pdf
T. Zerrouki. (2019, October 2022). Tashaphyne, Arabic light stemmer. Available: https://pypi.org/project/Tashaphyne/0.3.4.1/
R. Rehurek and P. Sojka, "Software framework for topic modelling with large corpora," in Proceedings of the LREC 2010 workshop on new challenges for NLP frameworks, Valletta, Malta, 2010, pp. 45-50.
C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Gérard-Marchant, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi, C. Gohlke, and T. E. Oliphant, "Array programming with NumPy," Nature, vol. 585, pp. 357-362, 2020/09/01 2020.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay, "Scikit-learn: Machine Learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.
C. E. Metz, "Basic principles of ROC analysis," Seminars in Nuclear Medicine, vol. 8, pp. 283-298, 1978/10/01/ 1978.
T. Pedersen, S. Patwardhan, and J. Michelizzi, "WordNet:: Similarity-Measuring the Relatedness of Concepts," in Proceedings of the Nineteenth National Conference on Artificial Intelligence (Sponsored by the AAAI), San Jose, California, USA, 2004, pp. 25-29.
C. Leacock and M. Chodorow, "Combining local context and WordNet similarity for word sense identification," in WordNet: An electronic lexical database. vol. 49, ed, 1998, pp. 265-283.
Z. Wu and M. Palmer, "Verbs semantics and lexical selection," in Proceedings of the 32nd annual meeting on Association for Computational Linguistics, Las Cruces, New Mexico, USA, 1994, pp. 133–138.
P. Resnik, "Using information content to evaluate semantic similarity in a taxonomy," in Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, Montreal, Quebec, Canada, 1995, pp. 448-453.
D. Lin, "An Information-Theoretic Definition of Similarity," in Proceedings of the Fifteenth International Conference on Machine Learning, 1998, pp. 296–304.
J. J. Jiang and D. W. Conrath, "Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy," in Proceedings of the 10th Research on Computational Linguistics International Conference, Taipei, Taiwan, 1997, pp. 19-33.
T. Berners-Lee. (2006, October 2022). Linked data-design issues. Available: https://www.w3.org/DesignIssues/LinkedData.html
J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P. N. Mendes, S. Hellmann, M. Morsey, P. van Kleef, S. Auer, and C. Bizer, "DBpedia – A large-scale, multilingual knowledge base extracted from Wikipedia," Semantic Web, vol. 6, pp. 167-195, 2015.
G. Hirst and D. St-Onge, "Lexical chains as representations of context for the detection and correction of malapropisms," in WordNet: An electronic lexical database. vol. 305, ed: MIT Press, 1998, pp. 305-332.
S. Banerjee and T. Pedersen, "Extended gloss overlaps as a measure of semantic relatedness," in Proceedings of the 18th international joint conference on Artificial intelligence, Acapulco, Mexico, 2003, pp. 805–810.
S. Patwardhan, "Incorporating dictionary and corpus information into a context vector measure of semantic relatedness (Doctoral dissertation, University of Minnesota, Duluth)," 2003.
Published
How to Cite
Issue
Section
Copyright (c) 2023 Journal of Information Technology and Computing
This work is licensed under a Creative Commons Attribution 4.0 International License.
- Copyright and Licensing
For all articles published in SABA journals, copyright is retained by the authors. Articles are licensed under an open access Creative Commons CC BY 4.0 license, meaning that anyone may download and read the paper for free. In addition, the article may be reused and quoted provided that the original published version is cited. These conditions allow for maximum use and exposure of the work, while ensuring that the authors receive proper credit.
