Intelligent Indexing Based on Usage Patterns and Query Frequency
DOI:
https://doi.org/10.63282/3050-9246.IJETCSIT-V4I2P113Keywords:
Intelligent Indexing, Query Frequency, Adaptive Indexing, Usage Patterns, Workload-Aware Databases, Big Data RetrievalAbstract
Since the age of database systems, web searches and big data systems, modern databases must support information search as a required function. In all the constantly multiplying mass of digital data can be seen, the traditionally very stable ways may find it difficult to keep up easily enough with the nature of its dynamics in the first place. The intelligent indexing be determined by use patterns and query frequency, which suggests an adaptive measure for query response time to increase, storage redundancy decrease and for user satisfaction to rise. The paper concerns a structured system of intelligent indexing which absorbs continuously query logs, access patterns and frequency distributions; then it attempts to rearrange its indices dynamically. This includes the probabilistic models, clustering by frequency, adaptive data structure that helps query processing to avoid the burden of high overhead. It is also illustrated by experimental simulations: compared to traditional indexing policies, intelligent indexing can lower average query latency by as much as 45 percent. On top of that the system also optimizes caching and provides workload-sensitive database tuning. Here work has added to the field of adaptive indexing both an approach from first principles and an actual evaluation of the current techniques with real query loads. The results confirm the usage patterns along with query frequency being part of index management is a powerful tool to enhance optimization for data retrieval systems with respect to scaling efficiency and response time
Downloads
References
[1] Kraska, T., Beutel, A., Chi, E. H., Dean, J., & Polyzotis, N. (2018, May). The case for learned index structures. In Proceedings of the 2018 international conference on management of data (pp. 489-504).
[2] Tekale, K. M., & Rahul, N. (2022). AI and Predictive Analytics in Underwriting, 2022 Advancements in Machine Learning for Loss Prediction and Customer Segmentation. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 3(1), 95-113. https://doi.org/10.63282/3050-9262.IJAIDSML-V3I1P111
[3] Idreos, S., Kersten, M. L., & Manegold, S. (2007, June). Updating a cracked database. In Proceedings of the 2007 ACM SIGMOD international conference on Management of data (pp. 413-424).
[4] Ding, J., Minhas, U. F., Yu, J., Wang, C., Do, J., Li, Y., ... & Kraska, T. (2020, June). ALEX: an updatable adaptive learned index. In Proceedings of the 2020 ACM SIGMOD international conference on management of data (pp. 969-984).
[5] Paludo Licks, G., Colleoni Couto, J., de Fátima Miehe, P., De Paris, R., Dubugras Ruiz, D., & Meneguzzi, F. (2020). SmartIX: A database indexing agent based on reinforcement learning. Applied Intelligence, 50(8), 2575-2588.
[6] Gu, T., Feng, K., Cong, G., Long, C., Wang, Z., & Wang, S. (2021). A reinforcement learning based r-tree for spatial data indexing in dynamic environments. arXiv preprint arXiv:2103.04541.
[7] Idreos, S., Manegold, S., Kuno, H., & Graefe, G. (2011). Merging what's cracked, cracking what's merged: adaptive indexing in main-memory column-stores. Proceedings of the VLDB Endowment, 4(9), 586-597.
[8] Luhring, M., Sattler, K. U., Schmidt, K., & Schallehn, E. (2007, April). Autonomous management of soft indexes. In 2007 IEEE 23rd International Conference on Data Engineering Workshop (pp. 450-458). IEEE.
[9] Licks, G. P., & Meneguzzi, F. (2020). Automated database indexing using model-free reinforcement learning. arXiv preprint arXiv:2007.14244.
[10] Tekale, K. M. (2022). Claims Optimization in a High-Inflation Environment Provide Frameworks for Leveraging Automation and Predictive Analytics to Reduce Claims Leakage and Accelerate Settlements. International Journal of Emerging Research in Engineering and Technology, 3(2), 110-122. https://doi.org/10.63282/3050-922X.IJERET-V3I2P112
[11] Comer, D. (1979). Ubiquitous B-tree. ACM Computing Surveys (CSUR), 11(2), 121-137.
[12] Chan, C. Y., & Ioannidis, Y. E. (1998, June). Bitmap index design and evaluation. In Proceedings of the 1998 ACM SIGMOD international conference on Management of data (pp. 355-366).
[13] Lim, Y., & Kim, M. (2004, August). A bitmap index for multidimensional data cubes. In International Conference on Database and Expert Systems Applications (pp. 349-358). Berlin, Heidelberg: Springer Berlin Heidelberg.
[14] Krčál, L., Ho, S. S., & Holub, J. (2021). Hierarchical Bitmap Indexing for Range and Membership Queries on Multidimensional Arrays. arXiv preprint arXiv:2108.13735.
[15] Chaudhuri, S., & Weikum, G. (2000, September). Rethinking Database System Architecture: Towards a Self-Tuning RISC-Style Database System. In VLDB (pp. 1-10).
[16] Halim, F., Idreos, S., Karras, P., & Yap, R. H. (2012). Stochastic database cracking: Towards robust adaptive indexing in main-memory column-stores. arXiv preprint arXiv:1203.0055.
[17] Wang, C., Zhu, Y., Ma, Y., Qiu, M., Liu, B., Hou, J., ... & Shi, W. (2018). A query-oriented adaptive indexing technique for smart grid big data analytics. Journal of Signal Processing Systems, 90(8), 1091-1103.
[18] Srihari, R. K., Zhang, Z., & Rao, A. (2000). Intelligent indexing and semantic retrieval of multimodal documents. Information Retrieval, 2(2), 245-275.
[19] Tekale, K. M. T., & Enjam, G. reddy . (2022). The Evolving Landscape of Cyber Risk Coverage in P&C Policies. International Journal of Emerging Trends in Computer Science and Information Technology, 3(3), 117-126. https://doi.org/10.63282/3050-9246.IJETCSIT-V3I1P113
[20] Chen, H. M., & Cooper, M. D. (2001). Using clustering techniques to detect usage patterns in a Web‐based information system. Journal of the American Society for Information Science and Technology, 52(11), 888-904.
[21] Bertino, E., Ooi, B. C., Sacks-Davis, R., Tan, K. L., Zobel, J., Shidlovsky, B., & Andronico, D. (2012). Indexing techniques for advanced database systems (Vol. 8). Springer Science & Business Media.
[22] Narang, R. (2018). Database management systems. PHI Learning Pvt. Ltd..
[23] Stupar, S., Bičo Ćar, M., Kurtović, E., & Vico, G. (2021, May). The importance of machine learning in intelligent systems. In International Conference “New Technologies, Development and Applications” (pp. 638-646). Cham: Springer International Publishing.
