Describes a data‑ingestion and caching mechanism using graph path cache structures to support low‑latency real‑time fraud or analytics pipelines

Authors

  • Jayaram Immaneni SRE LEAD at JP Morgan Chase, USA. Author

DOI:

https://doi.org/10.63282/3050-9246.IJETCSIT-V6I2P109

Keywords:

Data Ingestion, Caching Mechanism, Graph Path Cache, Low Latency, Fraud Analytics, Real-Time Pipelines

Abstract

In today's world of rapid fraud detection and analytics, systems need to be able to process and respond to large amounts of data in milliseconds. However, traditional data intake and caching pipelines don't meet these ultra-low-latency needs because they read data in order, do the same computations twice, and don't store data in the best place. These limitations make decision-critical systems, such as payment monitoring or behavioral analytics, have unnecessary bottlenecks. Even a delay of a millisecond might impair accuracy or risk reduction. To fix this problem, we provide a mechanism to store and use data that uses a graph route cache structure. This structure shows information & cache links as traversable graph routes instead of flat or hierarchical caches. This method treats each piece of information as a node that is linked to many other nodes by context-aware connections. This makes predictive & dependency-based caching possible. The ingestion system skillfully matches incoming data streams to this graph, which makes it easy to look things up & keeps the cache up to date without having to recalculate. When the latest data point comes in, the system finds the right graph paths & only gets and updates the nodes that are affected, instead of reloading complete datasets. Graph-based caching in the intake layer makes it easier to get their information that changes depending on the context & makes sure that the system stays in sync with changing actual time inputs. Tests show that this the latest cache design cuts query latency by 40% & doubles performance compared to previous cache setups. This technology combines fast intake with smart caching to create their scalable, low-latency data pipelines that may be used for more continuous fraud detection, actual time analytics & adaptive machine-learning processes

Downloads

Download data is not yet available.

References

[1] Murarka, Sachin, Anshuj Jain, and Laxmi Singh. "Advanced Techniques in Data Ingestion and Pipelining for Scalable Big Data Platforms: A Comprehensive Review." 2024 IEEE 4th International Conference on ICT in Business Industry & Government (ICTBIG). IEEE, 2024.

[2] Anisetti, Marco, et al. "Data analytics and ingestion-time access control initial report."

[3] Abeyratne, Dilshan. "Real-Time Streaming Analytics and Latency Minimization in Autonomous Vehicle Big Data Pipelines." Northern Reviews on Smart Cities, Sustainable Engineering, and Emerging Technologies 9.11 (2024): 49-62.

[4] Marcu, Ovidiu-Cristian, et al. "Storage and Ingestion Systems in Support of Stream Processing: A Survey." (2018): 1-33.

[5] Noman, Noman. "A Comparative Analysis of Modern Data Ingestion Platforms for Real-Time Processing Applications." (2025).

[6] D'Armiento, Alessandro. A distributed framework for real-time ingestion of unstructured streaming data. Diss. Politecnico di Torino, 2018.

[7] Marcu, Ovidiu-Cristian. KerA: A Unified Ingestion and Storage System for Scalable Big Data Processing. Diss. INSA Rennes, 2018.

[8] Marcu, Ovidiu-Cristian. KerA: A Unified Ingestion and Storage System for Scalable Big Data Processing. Diss. INSA Rennes, 2018.

[9] Rucco, Chiara, Antonella Longo, and Motaz Saad. "Enhancing Data Ingestion Efficiency in Cloud-Based Systems: A Design Pattern Approach." Data Science and Engineering (2025): 1-16.

[10] Barrios, Carlos, and Mohan Kumar. "Service caching and computation reuse strategies at the edge: A survey." ACM Computing Surveys 56.2 (2023): 1-38.

[11] Doherty, Conor, and Gary Orenstein. "Building Real-Time Data Pipelines." (2015).

[12] Atrushi, Diler, and Subhi RM Zeebaree. "Distributed Graph Processing in Cloud Computing: A Review of Large-Scale Graph Analytics." The Indonesian Journal of Computer Science 13.2 (2024).

[13] Olaoye, Godwin, Samuel Johnson, and Moses Blessing. "Batch to Real-Time: Leveraging AI for Streaming ETL Pipelines." (2025).

[14] Gupta, Sumit. "Real-Time Big Data Analytics." (2016).

[15] Raj, Pethuru, et al. "High-performance big-data analytics." Computing Systems and Approaches (Springer, 2015) 1 (2015).

Published

2025-05-05

Issue

Section

Articles

How to Cite

1.
Immaneni J. Describes a data‑ingestion and caching mechanism using graph path cache structures to support low‑latency real‑time fraud or analytics pipelines. IJETCSIT [Internet]. 2025 May 5 [cited 2025 Dec. 15];6(2):70-7. Available from: https://www.ijetcsit.org/index.php/ijetcsit/article/view/496

Similar Articles

11-20 of 369

You may also start an advanced similarity search for this article.