Add FSTools with examples of how to convert between SPIFFS and LITTLEFS.

[sticilface]

This lib provides a simple way, with some caveats, to 'in the field' conversion of a SPIFFS partition to LITTLEFS of different sizes.

The main caveat is the total size of all the files is limited to the gap between the end of the sketch and the start of the partition you are trying to copy.

I've created layout objects for all the possible layouts, these are in the namespace FST, however, they are global and are ALL linked with is a bit annoying. If any C++ gurus know how to create globals that can be dropped if they are not used, or has any other bright ideas, then let me know!

The example sketch basically tries to mount the new FS and fails, then the lib kicks in does the conversion and leaves the new FS mounted.

There is support for custom partitions as well if you are trying to roll an intermediate sketch and need to define your own rules.

[earlephilhower]

To fix the style errors, please run the tests/restyle.sh script and commit the changes. We enforce the Arduino format rules on the libs.

For the other build issues, we use -Wall -Werror. Easiest thing to do is use the IDE File->Preferences->Compile Warnings->All and rebuild to locate and then manually fix any sign/unused/etc. problems.

[sticilface]

Ah i wondered about the style fail, and there was no comment as to why.
The unused variable is in FS.h not my addition, i've removed it.
See how we get on...

[sticilface]

unfortunately if i pull the latest git to work with nothing works in my arduino IDE. I'm now fixing errors that didn't show up using error all in the IDE but relate to FS.h so not my addition... but all passed this time!

[earlephilhower]

unfortunately if i pull the latest git to work with nothing works in my arduino IDE. I'm now fixing errors that didn't show up using error all in the IDE but relate to FS.h so not my addition... but all passed this time!

Yeah, it looks like none of our example code (used in CI) was exercising that bit of code in FS.h you had to change. Good catch, and thx for fixing it.

Unfortunately, I don't understand the other comment. Is this worth testing, or is is still a WIP ("nothing works")?

[sticilface]

It is worth testing. My other comment is just referring to my ability to test source code from github either via PIO or the arduino IDE as for various reasons they all error. not released to this specific pull request.

[earlephilhower]

Very ingenious way of handling the FS conversion! Just a few minor things to clean up. Thx!

[sticilface]

Awesome, I will try and resolve all these points.

[sticilface]

See how these get on. Its taken more time to fix 'git' issues with merging than it did to fix the issues. I feel like my brain is bleeding.

[earlephilhower]

Please think about the yield() comment, but it looks good to me for a first commit. I think docs would be a good follow-on to help understand how to use it in users' own code, but I think for people who care about doing this, they should be able to figure it our from the examples.

[devyte]

If I understand this correctly, this works as follows:

1. create a secondary FS in the empty flash area
2. copy files over from the main FS to the secondary FS
3. reformat/mount primary FS
4. copy files over from secondary FS to newly created primary FS

In the above there is one very dangerous thing that I don't see considered: reset/power loss. If during steps 3/4 there is a reset or power loss, you lose the primary FS and have no state to remember, so you end up with a dead FS and no realistic way to recover.

Before an attempt is made to address that here with some custom fix, I suggest considering a different approach: create a FS image in the empty area and then figure out a way to trigger the FS update on next boot with the existing update mechanism like Updater does. That would streamline the update process with what we already know works, including existing support for power loss as well as future upcoming changes.

[devyte]

@earlephilhower pointed out to me a limitation in implementing the approach I outlined above: the secondary fs size is limited to the size of the empty area, and therefore the new FS is as well, which may not be desirable if the original fs was big but had very low usage.
Therefore, consider a custom fix for this pr
Implement in the examples e. g. setting an eeprom bit to survive a power loss/reset, then on boot up check it and if set jump straight to a retry of the fs copy. Once successful, clear the bit. In addition, an explanation of the danger, and how it is addressed, is needed somewhere, e. g. in the example itself. This is the minimum for my approval of this pr, the danger of not having a recovery mechanism is very real.
Then, I suggest implementing the outline above to update on reboot with the existing update mechanism in a separate pr. That approach is probably better suited for 4MB+ systems, and may simplify the sketch.

[sticilface]

@devyte Thank you for your feedback, this is a very good point that you raise.

I will need to think about this some more. I think the penny just dropped for the ATOMIC_FS flag though. Does this use the OTA area similar to what I am doing and then signal to the bootloader to copy it on reboot to the actual FS area.

If this is the case how does it cope with FS that are bigger than the ota area... or does it not and it just fails. or.... can it decompress an image... if so are there compression tools available... ie possible to compress the files when stored in the OTA area...

The boot flag bits are set in the RTC no, so an overt power fail would not preserve the state... I seem to remember the first 76 bytes of RTC memory are used for this purpose?

[sticilface]

Having dug a little deeper I should be able to use
https://github.com/earlephilhower/uzlib/tree/80765a42d657fcd3a24e036ac9822f9dbf862449
which is located in /tools/sdk/uzlib to compress files

[earlephilhower]

Having dug a little deeper I should be able to use
https://github.com/earlephilhower/uzlib/tree/80765a42d657fcd3a24e036ac9822f9dbf862449
which is located in /tools/sdk/uzlib to compress files

Unfortunately, I doubt that there is enough free RAM to actually do gzip compression. Just to decompress requires over 36K free RAM (not a problem in the bootloader, but not likely in a running app).

You don't need to set atomic_fs_update if you want the bootloader to copy over the new FS from one spot to another, just set the proper copy command addresses and it will work on reboot using the same method that the ATOMIC_FS_UPDATE flag uses. But I'd say that is the wrong thing to do and won't work.

If you rely on eboot to copy, in any way, that means your replacement FS total size would be limited in size to whatever it was generated as (i.e. <400KB). You can't just extend LittleFS or SPIFFS FSes like XFS or EXT4. Plus, the core would blindly use the full 1MB/2MB/3MB FS the code was compiled to expect, but your binary FS image would not be that size leading to very bad things.

Instead a sequence like:
0. Clear EEPROM flag

1. Create tempFS in free space
2. Copy from mainFS to tempFS
3. Set EEPROM flag (i.e. if unexpected reboot, your setup() should check for this flag and if set jump back to step 4 before booting main app)
4. Format mainFS to new filesystem, giving you correct FS of full 1-3MB size
5. Copy from tempFS to mainFS
6. Clear EEPROM flag
7. Done!

If your app reboots/loses power before 3) then no FS changes were done and you can start from scratch or leave the old FS as-is.
If you app reboots/loses power from 3)-6) then the mainFS should be assumed garbage and tempoFS should be assumed to exist
If your app reboots/loses power after 6), well the FS is already updated. :)

[sticilface]

thank you @earlephilhower that does make a lot of sense and tbh i've not currently got the time to dive deep into trying to get that to work!

The EEPROM idea sounds solid. I will implement something. There is likely to be a bit of a delay, I'm currently studying for some exams which take priority atm.

[devyte]

This is not a breaking change, removing 3.0.0 milestone.

[d-a-v]

secondary FS locations could be computed by boards.py script.
approving, this can be done once merged

[d-a-v]

@sticilface do you think you can run the new tests/restyle.sh to pass CI ?
clang-format-13 must be installed on your computer to do so, or one/I can PR on your PR for this fix if you prefer

[sticilface]

I'm very sorry, I just saw this. I'm afraid life got in the way, new job, moved house and have more professional exams so I've not touched programming since November and I'm unlikely to till next year. Thank you for finishing and merging, much appreciated.