-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
qctools Version 7.3 for Windows is crashing or not responding #218
Comments
Thanks Glenn. Can you run |
I'll try running the ffmpeg tomorrow, but it crashes before anything comes up on the screen, so before graphing. |
Also you could try opening QCTools, go to the preferences and deselect some of the filters, then open the file. Possible that the combination of filter and file is the issue. |
Hi, this is Ian Sundstrom, working with Glenn at MDPI. I ran a .mkv that cannot open in QCTools through FFMPEG, and it completed without any errors. Here is a report log from running the file through FFMPEG: |
Extract of log:
|
Crash is not reproducible on my Windows 10 x64. Is it possible to produce crash dump or provide me with sample video causing crash ? |
Here are the steps for producing crash dump:
|
Thanks for looking into this. Here's some additional information: The files we are QCing are 45Gb to 95Gb (1.5 hours to 4 hours in content length)in size, possibly making it difficult to provide you with a sample file, in addition to us not being permitted to share the files outside of our lab. More often than crashing, the program just won't respond. We've never been able to successfully load anything with a file size that is much larger then 40Gb. This has been consistent with our experience concerning large files we've encountered in the past with other formats, as well. We are now into a large run of VHS files, most of which are 2hrs or longer(60Gb-95Gb). We are unable to use QCTools to QC these files. It is our hope that in the near future QCTools will be able to accommodate larger file sizes. I'm not sure if theses two issues are related. My associate, Ian, will run a crash dump on one of our offending files. Hopefully by the end of today. Glenn Hicks From: elderorb [email protected] Crash is not reproducible on my Windows 10 x64. Is it possible to produce crash dump or provide me with sample video causing crash ? You are receiving this because you authored the thread. |
Here's a procdump for a different file. The file I used for the FFMPEG log wouldn't cause QCTools to crash today (it would simply not respond to being dragged into the program), and thus couldn't produce a procdump, so I opted for one that was causing a crash. Here is the file: https://www.slashtmp.iu.edu/files/download?FILE=isundstr%2F96274TJlbiI |
This command provides a close sample to @iwsundstrom except I'm not exactly sure how to replicate the 1/1000 timebase that file uses.
Increase Also @glwhicks @iwsundstrom can you confirm what settings you have in your preferences (what filter and first/all tracks). |
@glwhicks You could probably just cut the first 100 MB off an example to replicate the issue. |
We usually only use signalstats and PSNR filters, however, when you suggested testing a variety of different ones, we did so and QCTools still always crashed. |
ok, i was suspicious of the audio, but apparently you're not even using that. |
I was wondering about that as well, Dave. However, we've never been able to see the audio graph populate with any video format. My theory is the .mkv format might be the issue. The VHS files have four channels of audio, 2 linear and 2 hi-fi. Might that be causing some problems, perhaps? However, I have a 47Gb(1hr 2mins) UMatic file with only two channels of audio that doesn't respond either. Glenn Hicks From: Dave Rice [email protected] ok, i was suspicious of the audio, but apparently you're not even using that. You are receiving this because you were mentioned. |
Thank you for the dump provided, looking at this. Could you please also check if the latest qctools snapshot behaves in the same way? |
@JeromeMartinez created a debug version of the Windows build at https://mediaarea.net/temp/QCTools_0.7.3.20161118_Windows_i386_WithDebugInfo.zip. This contains the exe and the corresponding pdb file. @glwhicks @iwsundstrom could you re-test with this version. |
I just tried the debugged version and the first file I dropped into it crashed it. It was a 55Gb .mkv file |
I'll give it a test on our Windows machine today and see if I get the same On Mon, Nov 21, 2016 at 8:00 AM, glwhicks [email protected] wrote:
Kelly Haydon | Preservationist Archivists can use the power of archives to promote accountability, open |
@glwhicks Could you please collect crash dump for debug build? Release dump didn't reveal any obvious reasons for crash.. |
@JeromeMartinez it is possible to produce 64bit build for Windows? Just thinking this random crashes could be 'out of memory' exception. |
I am loading a 40gig ffv1/mkv video file into the debugged Windows build Audio seems to be loading normally. Not sure if this is relevant, but I do have an MDPI qctools report that On Mon, Nov 21, 2016 at 1:47 PM, elderorb [email protected] wrote:
Kelly Haydon | Preservationist Archivists can use the power of archives to promote accountability, open |
We already provide 64-bit builds, look at "Windows_x64" in the snapshots. For official release, the Installer has actually only 32-bit version despite the "Universal" name, but the "64 bit only without installer" is the pure 64-bit version.
As it is a the start of the parsing, and also a 0x00000004 memory access, I bet more on a NULL pointer not tested somewhere. |
Could be... But so far I managed to reproduce crash only in two cases:
|
@kellyhaydon can you share the mdpi xml? |
I'm not sure I can here - @glwhicks https://github.com/glwhicks? But the file is in our shared "qctools 2.0" google doc. On Mon, Nov 21, 2016 at 4:49 PM, Dave Rice [email protected] wrote:
Kelly Haydon | Preservationist Support BAVC today. * Archivists can use the power of archives to promote accountability, open |
My associate Ian, will be in tomorrow to provide the requested crashdump. In the meantime, here is the requested XML. In addition I'm providing the QCTools report generated by Memnon. Perhaps this might provide some insight: https://slashtmp.iu.edu/files/download?FILE=glwhicks%2F15154X03p67 |
To be clear, the provided XML and QCTools report are from a file that crashed the program |
Does
mean
? |
Just tried 32bit build under debugger - now I'm 100% sure this is out of memory. First qctools allocated 564029759 bytes for a buffer to extract xml.gz file, and then it tried to create std::string from buffer (which means allocation of at least 564029759 one more time) - and this time allocation failed. @dericed & @JeromeMartinez what is the reason for not producing 64bit installers for windows? |
Yes, Dave, correct. Never with a neighboring xml.gz file. |
@glwhicks just try x64 version from here - https://mediaarea.net/download/snapshots/binary/qctools/20161204/ - should fix the issue |
Okay! Thanks so much for all the great input! |
Welcome! but please keep in mind the link provided is not released version, but development snapshot. |
There is still some 32-bit Windows there (Microsoft did not "kill" 32-bit as Apple did for macOS, and there is unfortunately a 32-bit version of Windows 10), so I provide the default installer in 32-bit in order to be compatible with all versions of Windows. Providing several .exe is sometimes a source of confusion for end user who does not know what is 32 and 64-bit, so this is a policy to provide the 64-bit version only in an archive. Edit: discussion moved to MediaArea/MediaArea-Utils#96 |
Unfortunately, I just ran the x64 version of QCTools and it crashed while trying to open the test file with it's xml.gz. Is there a debug version of x64 QCTools that I can use to send another crash report? |
In addition, we just tested moving the xml.gz file out of the folder and opening our large test file. It reached completion and generated all the graphs without any problems. We then decided to test "Export->To .qctools.xml.gz" to create a new xml.gz file. Our plan was to close QCTools and open the test file with the freshly generated xml. However, QCTools crashed when we tried to do the export. Not sure if this is helpful, but thought we should pass it along. |
Due to a problem when we applied our patches for release to this branch. |
Great, thanks. Here's the crash dump produced with that version: |
Just checked version @JeromeMartinez provided - and managed to open MDPI_40000001519547_01_pres.mkv.qctools.xml.gz with no crash on Windows 10. @iwsundstrom Did you use the same xml.gz file @glwhicks posted ? If different - could you please provide that file? Also, just to clarify, was it crash on export or on import? |
With the build at #218 (comment), I have sometimes a "Out of memory! Please try 64bit build." message despite the fact I run the 64-bit version, or a crash, depending of how loaded (RAM usage) is my machine. |
@ElderOrb Same file with xml.gz, and the crash is from an import with the x64 debug version provided by Jerome. The "export" story was just an additional note that I thought might be helpful. |
@iwsundstrom How much memory do you have on this machine? While switching to 64bit could help (just because it allows to use more than 4Gb), it will not help if you don't have enough memory or memory is heavily fragmented. QCTools definitely requires some optimizations and @JeromeMartinez already started optimizing xml parsing. So maybe makes sense to check his version if he can provide a link |
@ElderOrb It has 16GB of RAM. I'm excited to test the optimized version as it does sound like XML parsing is problem. |
How much free memory do you have before opening xml.gz and after getting crash? Besides the lack of memory it might also be related to memory fragmentation, which is different story. Also, you have video file, correct? Which mean as the result of opening xml.gz thumbnails being generated and they also can consume a lot. Could you please make simple experiment and move/rename video file and then open xml.gz last 64bit debug version? This should remove influence of thumbnails and possibly resolve the crash |
In regards to how much memory is used during a crash: Before I open a file I am using about 3.4 GB. During opening the memory builds up to about 9.8 GB used max, wavering up and down between there and lower. This is while opening the video file with the XML.gz beside it. As for the test of just opening the XML.gz (without the video or thumbnails), that worked. The file successfully opened and populated the graphs, although QCTools was "Not Responding" for a short while. I'm wondering if including some sort of loading or progress bar may be a good feature in the future, beyond resolving the crashes. It is not very user friendly to open large files and not be given feedback on loading progress. |
This is already the case when there is no sidecar .gz file (graph is populated in another thread), it could be implemented also for loading the .gz, the proposed patch is in this direction (there is some "hooks" for being able to display some info to the user), but having the progress bar has IMHO some impact in the workflow, it is more development time :(. |
We can leave importing in the UI thread and add progress bar (QApplication::processEvents trick). And I absolutely agree, during importing huge files progress bar is 'must have' thing |
I guess it is a good first step. |
Sad to report that the new snapshot still crashes. Here's a crash dump: |
Based on what I see in the dump, this time 'access violation' happened inside 'ModifyOutput' function, which is used during video parsing, so this is not lack of memory but something different. Unfortunately I don't have that video (and don't even have such a big video right now), @dericed any help on generating such a long video with the same characteristics? Also I noticed you still have application verifier enabled, could you please disable it (in very rare cases it might produce false positives) and see if application still crashes? Also just to exclude lack of memory (although as I said, it is highly unlikely this issue is related to allocations), would be interesting to see how x64 version will behave. But for this we need ask @JeromeMartinez to produce 64bit build from his branch AVRF INVALID_POINTER_READ BUGCHECK_STR: INVALID_POINTER_READ_AVRF DEFAULT_BUCKET_ID: INVALID_POINTER_READ_AVRF STACK_TEXT: STACK_COMMAND: .ecxr ; kb ; dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; dps 27cbd8 ; kb THREAD_SHA1_HASH_MOD_FUNC: 149eb2a0da52c8b38c064648615db142b8fd336c THREAD_SHA1_HASH_MOD_FUNC_OFFSET: b58a2edb0e3a82915bf0f8a0aee65193bc7c7531 THREAD_SHA1_HASH_MOD: 52ccbee68d27742fc5b25097be25e66065a90a48 FAULT_INSTR_CODE: 13c83 FAULTING_SOURCE_LINE: c:\programmation\build_7220\qctools_allinclusive\qctools\source\core\ffmpeg_glue.cpp FAULTING_SOURCE_FILE: c:\programmation\build_7220\qctools_allinclusive\qctools\source\core\ffmpeg_glue.cpp |
I disabled Application Verifier and was able to open the test file very quickly. I retried it multiple times (closing QCTools and opening it again) and it worked every time. I also tried using the snapshot to open other long videos that haven't worked with previous versions and was successful there to. It appears the issue has been fixed and that Application Verifier caused the previous crash! Thanks for working with us on this issue. We really appreciate it! |
Closing for now as this seems resolved. Re-open if needed. |
Hi all! AMIA was fantastic! Great to see the qctools presentations and meet everybody. I downloaded Version 7.3 for Windows before I left for Pittsburgh on our machines and was eager to get get started Qcing with my new found knowledge, but alas, most of the press files(.mkv) have caused the program to either crash or simply not respond. I did get one file to open. We use: Dell computers running Windows 7 Enterprise 64-bit Operating System, 16GB RAM, Intel(R) Xeon(R) CPU E5-1620 v3 @3.50GHz
The text was updated successfully, but these errors were encountered: