Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data loss when interrupting import #4283

Closed
hamaryns opened this issue Feb 17, 2022 · 11 comments
Closed

Data loss when interrupting import #4283

hamaryns opened this issue Feb 17, 2022 · 11 comments
Labels
needinfo We need more details or follow-up from the filer before this can be tagged "bug" or "feature."

Comments

@hamaryns
Copy link

hamaryns commented Feb 17, 2022 •
edited
Loading

I am importing my music collection for the first time. At a certain moment I had to interrupt the proces. That took a while, so I hit Ctrl-c. Now, if I restart, it resumes, but after a while a song is not found in the new location where beets expects it, and it is also no longer in the old location. Data loss!

Problem

Running this command in verbose (-vv) mode:

$ beet import /media/me/external-usb/music to be sorted

It is still running in verbose mode…

Led to this problem:

This album is already in the library!
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Kryptonite.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Loser.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Duck and Run.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Not Enough.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Be Like That.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Life of My Own.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 The Better Life.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Down Poison.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 By My Side.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Smack.flac'
could not get filesize: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 So I Need You.flac'
Old: 11 items, FLAC, 1631kbps, 44.1kHz/24 bit, 40:51, 0.0 B
Old: 11 items, FLAC, 1631kbps, 44.1kHz/24 bit, 40:51, 476.3 MiB
New: 1 items, FLAC, 1644kbps, 44.1kHz/24 bit, 3:07, 36.8 MiB
[S]kip new, Keep all, Remove old, Merge all? m
Traceback (most recent call last):
  File "/home/hendrik2/.local/bin/beet", line 8, in <module>
    sys.exit(main())
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/ui/__init__.py", line 1285, in main
    _raw_main(args)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/ui/__init__.py", line 1272, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/ui/commands.py", line 973, in import_func
    import_files(lib, paths, query)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/ui/commands.py", line 943, in import_files
    session.run()
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/importer.py", line 340, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/util/pipeline.py", line 446, in run_parallel
    raise exc_info[1].with_traceback(exc_info[2])
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/util/pipeline.py", line 314, in run
    for msg in _allmsgs(out):
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/util/pipeline.py", line 467, in pull
    out = coro.send(msg)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/util/pipeline.py", line 193, in coro
    func(*(args + (task,)))
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/importer.py", line 1369, in lookup_candidates
    plugins.send('import_task_start', session=session, task=task)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/plugins.py", line 488, in send
    result = handler(**arguments)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beets/plugins.py", line 145, in wrapper
    return func(*args, **kwargs)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beetsplug/chroma.py", line 180, in fingerprint_task
    return fingerprint_task(self._log, task, session)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beetsplug/chroma.py", line 249, in fingerprint_task
    acoustid_match(log, item.path)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/beetsplug/chroma.py", line 89, in acoustid_match
    duration, fp = acoustid.fingerprint_file(util.syspath(path))
  File "/home/hendrik2/.local/lib/python3.8/site-packages/acoustid.py", line 340, in fingerprint_file
    return _fingerprint_file_audioread(path, maxlength)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/acoustid.py", line 280, in _fingerprint_file_audioread
    with audioread.audio_open(path) as f:
  File "/home/hendrik2/.local/lib/python3.8/site-packages/audioread/__init__.py", line 111, in audio_open
    return BackendClass(path)
  File "/home/hendrik2/.local/lib/python3.8/site-packages/audioread/rawread.py", line 62, in __init__
    self._fh = open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: b'/media/hendrik2/Scharnier/Muziek/3 Doors Down/The Better Life/3 Doors Down \xe2\x80\x93 Kryptonite.flac'

and then command line.

Here's a link to the music files that trigger the bug (if relevant):
it is on a local external usb drive

Setup

  • OS: Linux Mint Mate edition 20.3 64 bit
  • Python version:
    $ python2 --version
    Python 2.7.18
  • beets version:
    $ beet --version
    beets version 1.6.0
    Python version 3.8.10
    plugins: chroma, fetchart, fromfilename, lastgenre
    (different python versions, that’s interesting)
  • Turning off plugins made problem go away (yes/no):

My configuration (output of beet config) is:

$ beet config
directory: /media/hendrik2/Scharnier/Muziek
library: ~/.beets/musiclibrary.db

import:
    move: yes
log: beetslog.txt
art_filename: "$albumartist \u2013 $album"

plugins: fetchart lastgenre chroma fromfilename

paths:
    default: "$albumartist/$album%aunique{}/$artist \u2013 $title"
    singleton: "$artist/$artist \u2013 $title"
    comp: "Compilations/$album%aunique{}/$artist \u2013 $title"
acoustid:
    apikey: REDACTED
fetchart:
    auto: yes
    minwidth: 0
    maxwidth: 0
    quality: 0
    max_filesize: 0
    enforce_ratio: no
    cautious: no
    cover_names:
    - cover
    - front
    - art
    - album
    - folder
    sources:
    - filesystem
    - coverart
    - itunes
    - amazon
    - albumart
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    lastfm_key: REDACTED
    store_source: no
    high_resolution: no
    deinterlace: no
    cover_format:
lastgenre:
    whitelist: yes
    min_weight: 10
    count: 1
    fallback:
    canonical: no
    source: album
    force: yes
    auto: yes
    separator: ', '
    prefer_specific: no
    title_case: yes
chroma:
    auto: yes

I tried removing .beets/musiclibrary.db, but the error seems to persist.
@sampsyo sampsyo added the needinfo We need more details or follow-up from the filer before this can be tagged "bug" or "feature." label Feb 18, 2022
@sampsyo
Copy link
Member

sampsyo commented Feb 18, 2022

That sounds bad! Is the original problem (the interrupted import that deletes files) something you can reproduce reliably? Can you get a verbose log from that?

@hamaryns
Copy link
Author

hamaryns commented Feb 20, 2022 •
edited
Loading

It is not easy to reproduce, no. It happened again later, but with another file.
Did a verbose try, attached is the redirected stderr, with -vv

`$ beet -vv import /media/hendrik2/Scharnier/Muziek\ te\ sorteren/ 2> beet.log
Import of the directory:
/media/hendrik2/Scharnier/Muziek te sorteren
was interrupted. Resume (Y/n)?

/media/hendrik2/Scharnier/Muziek te sorteren/An Pierlé/Helium Sunset (+ enkele andere) (3 items)
Finding tags for album "An Pierlé - Helium Sunset (+ enkele andere)".
Candidates:

  1. An Pierlé - Sing Song Sally (50.3%) (tracks, album, year) (CD, 2003, BE, Helicopter, 0927458945)
  2. An Pierlé - As Sudden Tears Fall (47.4%) (tracks, album) (CD, 2002, BE, Helicopter, PRO3010)
  3. An Pierlé - Helium Sunset (46.5%) (missing tracks, mediums, album, ...) (2xCD, 2004, FR, [PIAS] France, PIASF 089 DCD)
  4. An Pierlé - Helium Sunset (43.7%) (missing tracks, tracks, album) (CD, 2002, BE, Helicopter, 0927 44423 2)
  5. An Pierlé - Arches (30.0%) (tracks, missing tracks, album, ...) (CD, 2016, BE, [PIAS] Belgium, 941.B438.020)

selection (default 1), Skip, Use as-is, as Tracks, Group albums,

Enter search, enter Id, aBort? 4
Correcting tags from:
An Pierlé - Helium Sunset (+ enkele andere)
To:
An Pierlé - Helium Sunset
URL:
https://musicbrainz.org/release/87f267b4-617d-4361-94bf-d9fdfb91a379
(Similarity: 43.7%) (missing tracks, tracks, album) (CD, 2002, BE, Helicopter, 0927 44423 2)

beet.log

The file is indeed nowhere to be found.

Another try, without plugins:
`$ beet -vv -plugins= import /media/hendrik2/Scharnier/Muziek\ te\ sorteren/ 2> beet.log
Import of the directory:
/media/hendrik2/Scharnier/Muziek te sorteren
was interrupted. Resume (Y/n)?

/media/hendrik2/Scharnier/Muziek te sorteren/An Pierlé/Helium Sunset (+ enkele andere) (3 items)
Finding tags for album "An Pierlé - Helium Sunset (+ enkele andere)".
Candidates:

  1. An Pierlé - Sing Song Sally (64.6%) (album, tracks, year) (CD, 2003, BE, Helicopter, 0927458945)
  2. An Pierlé - As Sudden Tears Fall (54.7%) (tracks, album) (CD, 2002, BE, Helicopter, PRO3010)
  3. An Pierlé - Helium Sunset (50.5%) (missing tracks, tracks, album) (CD, 2002, BE, Helicopter, 0927 44423 2)
  4. An Pierlé - Helium Sunset (45.9%) (missing tracks, mediums, album, ...) (2xCD, 2004, FR, [PIAS] France, PIASF 089 DCD)
  5. An Pierlé - Arches (36.0%) (missing tracks, tracks, album, ...) (CD, 2016, BE, [PIAS] Belgium, 941.B438.020)

selection (default 1), Skip, Use as-is, as Tracks, Group albums,

Enter search, enter Id, aBort? 4
Correcting tags from:
An Pierlé - Helium Sunset (+ enkele andere)
To:
An Pierlé - Helium Sunset
URL:
https://musicbrainz.org/release/bab2eed8-c228-4f97-9600-5db8239a013d
(Similarity: 45.9%) (missing tracks, mediums, album, tracks) (2xCD, 2004, FR, [PIAS] France, PIASF 089 DCD)
CD 2

/media/hendrik2/Scharnier/Muziek te sorteren/An Pierlé/Helium Sunset (+ enkele andere)
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [52]/An Pierlé – Sister.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Sorry.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – As Sudden Tears Fall.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Nobody's Fault.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Sister.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Kiss Me.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Helium Sunset.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Medusa.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Once Again.flac
/media/hendrik2/Scharnier/Muziek te sorteren/An Pierlé/Helium Sunset/An Pierlé – Nummer 14.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Leave Me There.flac
/media/hendrik2/Scharnier/Muziek/An Pierlé/Helium Sunset [53]/An Pierlé – Walk.flac (15 items)
Tagging:
An Pierlé - Helium Sunset
URL:
https://musicbrainz.org/release/87f267b4-617d-4361-94bf-d9fdfb91a379
(Similarity: 94.0%) (unmatched tracks) (CD, 2002, BE, Helicopter, 0927 44423 2)
Unmatched tracks (4):
! Il est cinq heures, Paris s’éveille (# 1) (4:00)
! 18 Nicotene (# 4) (3:06)
! Sing Song Sally (# 4) (4:11)
! Sister (# 4) (3:04)
[A]pply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort?

/media/hendrik2/Scharnier/Muziek te sorteren/Andrew Bird/Oh! The Grandeur (1 items)
Finding tags for album "Andrew Bird - Oh! The Grandeur".
Candidates:

  1. Andrew Bird - Oh No (50.0%) (album, tracks, missing tracks, ...) (CD-R, 2009, GB, Bella Union)
  2. Andrew Bird - Oh No (46.9%) (album, tracks, year) (Digital Media, 2008, US, Fat Possum Records)
  3. Andrew Bird - Oh No (45.8%) (album, missing tracks, tracks, ...) (CD-R, 2009, US, Fat Possum Records, [none])
  4. Andrew Bird’s Bowl of Fire - Oh! The Grandeur (34.4%) (missing tracks, artist) (Digital Media, 2010, XW, Rykodisc)
  5. Andrew Bird’s Bowl of Fire - Oh! The Grandeur (34.4%) (missing tracks, artist) (CD, 1999, US, Rykodisc, RCD 10398)

selection (default 1), Skip, Use as-is, as Tracks, Group albums,

Enter search, enter Id, aBort? 5
Correcting tags from:
Andrew Bird - Oh! The Grandeur
To:
Andrew Bird’s Bowl of Fire - Oh! The Grandeur
URL:
https://musicbrainz.org/release/dd26a699-007a-414b-8d5b-2a6c48c2a5f8
(Similarity: 34.4%) (missing tracks, artist) (CD, 1999, US, Rykodisc, RCD 10398)

beet2.log

(Something going wrong with formatting here, tried fixing it, but to no avail.)

@sampsyo sampsyo mentioned this issue Feb 21, 2022
Closed
@hamaryns
Copy link
Author

hamaryns commented Mar 5, 2022

another case, see log
beet.log

@sampsyo
Copy link
Member

sampsyo commented Mar 5, 2022

I'm really sorry, but there's not much we can do without a way to reproduce the problem.

@hamaryns
Copy link
Author

I have figured out a pattern. Unsure whether this is the same problem, but it looks similar.
What happens: during importing, I notice I made a mistake. I then move the imported folder back to the ‘to be sorted out’ folder. At a later run, when beets tries to tag this folder, I of course get a message that the files are already in the database, with something like this:

Old: 38 items, FLAC, 812kbps, 44.1kHz/16 bit, 145:44, 0.0 B
New: 38 items, FLAC, 812kbps, 44.1kHz/16 bit, 145:44, 845.5 MiB
[S]kip new, Keep all, Remove old, Merge all?

Notice the file size of the ‘old’ files, which are indeed no longer there. If I then choose M, it crashes after a while. Attached is the log.
beet31.log

The obvious workaround is to delete the database file and re-import everything.

@wisp3rwind
Copy link
Member

wisp3rwind commented Apr 19, 2022 •
edited
Loading

The obvious workaround is to delete the database file and re-import everything.

While I can't tell what a brief glance what the actual issue could be; the obvious workaround here should rather be to first use beets remove without -d to remove the album from the database, and then move and re-import as you did. Or, depending on the kind of mistake you made, reimporting via beet import -L might also work.

@arogl
Copy link
Contributor

arogl commented Apr 19, 2022

As you know you moved the files, it is safe to use the Remove old option.

Beets will then

  • delete the files in the DB
  • if the directory is empty, beets will delete the directory
  • go through the standard import routine

If you need a different music ID (muscbrainz, discogs, etc) you will have to either clear the exiting tags or pass an ID on the command line

@hamaryns
Copy link
Author

remove old works, thanks. I could not find that documented?

@wisp3rwind
Copy link
Member

We might still consider making this a safer experience: Merge probably isn't the most appropriate action to choose in this case, but maybe it shouldn't crash the importer. Things we could do

  • If all files of one of the duplicate albums are missing, don't even offer to merge,
  • If only some of the files are missing, inform the user and remove the corresponding items from the library and the import task. The current problem is that duplicate merging involves a re-import (see the user_query import stage), which would then involve an import task with partially missing files. Our importer is probably in general not expecting this. This manifests here for the chroma plugin, since it runs very early. However, I strongly suspect that other plugins and even core beets would lead to a crash due to missing files, even if chroma is disabled.

Note that in any case, this is about using beets out of spec: Beets for the most part assumes that it's in exclusive control of the library (including associated files). So we might decide that none of the above is really worth the effort (I won't be the one to implement this anyway...).

@hamaryns
Copy link
Author

hamaryns commented May 3, 2022

Well, might I suggest at least a more meaningful error message? Something like: ‘The file X in you library was not found, it is probably moved or deleted on the file system. Use “beet remove old” to update your library.’

@hamaryns hamaryns reopened this May 3, 2022
@wisp3rwind wisp3rwind mentioned this issue May 6, 2022
Open
@wisp3rwind
Copy link
Member

Closing in favor of #4342, which is probably the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
needinfo We need more details or follow-up from the filer before this can be tagged "bug" or "feature."
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@sampsyo @arogl @hamaryns @wisp3rwind