Hi,
for a long time I have observed that when I run some RTMaps diagrams, sometimes there are several records in the end of the recording corrupted, the replay can still finish but loading the recording in the Player can take a lot of time.
Does this happen when the recording is stopped unproperly like upon power loss of the datalogger?
Is there a way to fix such a corrupt dataset?
Hi LiL,
You’re right: in case of power loss for instance, or a software crash, and while the RTMaps diagram is recording, then the files written to disk are not closed property.
The data written at the end of each file may end-up being corrupt, and the consistency check between the .rec file and the .idx file will fail.
(Note: the .idx file is an index file that allows to perform almost instantaneous random access in your recording at replay time, that means jumping to a certain point in time in the dataset is performed in an optimized way, not dependent on the size nor the duration of the recording).
When loading such a corrupt recording in the Player component, it will report the following warning message in the RTMaps console:
“Warning: .idx file doesn’t match .rec file. No direct access index available.”
Loading the recording in the Player may then take long (as it has to parse the .rec file entirely to find the last timestamp without taking advantage of the .idx file), and any jump in time during replay for accessing a certain position in time, could also take time (as the Player has to parse all the .rec file lines from where it was replaying and to the destination time).
Additionally, some data samples at the end of the recording may have been lost since the last chunks of data would be buffered in the OS storage cache or disks buffers when the process stopped without gracefully closing the files.
Good news: most of the recorded data (up to the last seconds or frames which may not have been physically flushed to disk) are still there and are not lost.
Also, it is possible to rebuild an .idx file and hence take advantage of instantaneous .rec file loading in the Player and instantaneous random access.
For rebuilding an .idx that is consistent with the corrupted .rec file, just use utility “rtmaps_idx_regenerator” which is available in the /bin subfolder in your RTMaps installation folder.
Would you like to be able to test if a certain recording is completely valid or partly corrupt at the end (and with an inconsistent .idx file) due to such an ungraceful closing during recording, you may try these few python lines in the attached script.
Call
“python recfile_integrity_checker.py [myrecfile.rec]”
It will look for the corresponding .idx file, check consistency, print the result and return:
recfile_integrity_checker.py (920 Bytes)