Skip to content

Commit

Permalink
fix memory crashes :D
Browse files Browse the repository at this point in the history
  • Loading branch information
TheLastGimbus committed Feb 8, 2023
1 parent 38ea053 commit e0d9ee3
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 1 deletion.
7 changes: 7 additions & 0 deletions lib/date_extractors/exif_extractor.dart
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,18 @@ import 'dart:io';
import 'dart:math';

import 'package:exif/exif.dart';
import 'package:gpth/utils.dart';
import 'package:mime/mime.dart';

/// DateTime from exif data *potentially* hidden within a [file]
///
/// You can try this with *any* file, it either works or not 🤷
Future<DateTime?> exifExtractor(File file) async {
// if file is not image or >32MiB - DO NOT crash :D
if (!(lookupMimeType(file.path)?.startsWith('image/') ?? false) ||
await file.length() > maxFileSize) {
return null;
}
final bytes = await file.readAsBytes();
// this returns empty {} if file doesn't have exif so don't worry
final tags = await readExifFromBytes(bytes);
Expand Down
6 changes: 5 additions & 1 deletion lib/media.dart
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import 'dart:io';

import 'package:crypto/crypto.dart';
import 'package:gpth/utils.dart';

/// Abstract of a *media* - a photo or video
/// Main thing is the [file] - this should not change
Expand Down Expand Up @@ -34,7 +35,10 @@ class Media {
Digest? _hash;

/// will be used for finding duplicates/albums
Digest get hash => _hash ??= sha256.convert(file.readAsBytesSync());
/// WARNING: Returns same value for files > [maxFileSize]
Digest get hash => _hash ??= file.lengthSync() > maxFileSize
? Digest([0])
: sha256.convert(file.readAsBytesSync());

Media(
this.file, {
Expand Down
3 changes: 3 additions & 0 deletions lib/utils.dart
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ import 'package:proper_filesize/proper_filesize.dart';
// remember to bump this
const version = '3.3.2';

/// max file size to read for exif/hash/anything
const maxFileSize = 64 * 1024 * 1024;

/// convenient print for errors
void error(Object? object) => stderr.write('$object\n');

Expand Down

5 comments on commit e0d9ee3

@matt-boris
Copy link
Contributor

@matt-boris matt-boris commented on e0d9ee3 Feb 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works like a charm! 🎉 Thanks so much!

Now if only we could get brendan-duncan/image#457 (comment), then it could get working with large videos too :)

@matt-boris
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory utilization never went above 30% on my 8GB system 😄

@matt-boris
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The best message :) Thanks again so much! 🎉

DONE! FREEEEEDOOOOM!!!
Skipped 36 duplicates
Couldn't find date for 5897 photos/videos :/
Copied 31442 files to "Photos/Google Photos"

@TheLastGimbus
Copy link
Owner Author

@TheLastGimbus TheLastGimbus commented on e0d9ee3 Feb 9, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very glad it works!

Couldn't find date for 5897

By the way, if you have such big sample, could you look at them and share their filenames? Especially ones that we could guess the date from - you could then use the --fix mode to set them correct 🤔

@matt-boris

Ps

working with large videos too :)

There is (usually) no exif in videos :( they do sometimes have some metadata, but it usually differs

@matt-boris
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TheLastGimbus here are some example filenames where it looks like the dates should be able to be extracted:

00004XTR_00004_BURST20190216172030.jpg
00055IMG_00055_BURST20190216172030_COVER.jpg
2016_01_30_11_49_15.mp4
2020??.mov (lol wut?)
201801261147521000.jpg
Facetune_14-02-2019-01-02-36.jpg
IMG_1_BURST20160623205107_COVER.jpg
IMG_1_BURST20160520195318.jpg
IMG_2400_ADMIN_Feb-08-173238-2023_Conflict.MP4

Please sign in to comment.