Streamline memory consumption #1

New Issue

2024-04-04T07:40:46Z

joshuacoles commented

2024-04-04 07:40:46 +00:00

		arc-ingester/src/main.rs
		Line 96 in 3581797ebe
	
				            let bytes = tokio::fs::read(&path).await.unwrap();

Currently we are reading all gzip files before comparing the hashes, this uses needlessly large amounts of memory, hashes can be filtered at the time of computation.

https://git.joshuacoles.me/joshuacoles/arc-ingester/src/commit/3581797ebe63c430a99c07cf68aceb404baa381c/src/main.rs#L96 Currently we are reading all gzip files before comparing the hashes, this uses needlessly large amounts of memory, hashes can be filtered at the time of computation.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: joshuacoles/arc-ingester#1