Using Bootloader to Read SD Card

[wuuff]

I was thinking about the fact that the bootloader requires access to the SD card in order to load new games, so I started wondering if it would be possible to take advantage of this existing code in a game to read from SD. I decided to go check out the source. I found that it is written so that it directly loads a game instead of having a function to read data from a file, so I know it cannot be used easily, and therefore some additional logic would have to be put in the game. However, I was wondering if it might be possible to directly call some of the functions that the bootloader uses to read the files, such as file_read_hex.

There are a lot of issues I can imagine with this. First of all, some important functions, such as fat16_init and fat16_readRootDirEntry are inlined, so there is no way to call them. In addition, fat16_readRootDirEntry reads the very bottom of the stack for the filename (load_game appears to clobber the bottom of the stack at the very end of RAM by writing the filename---would these last 9 bytes ever be read once a game starts?), and always indicates failure if the file does not end in HEX or EEP.

Also, there is a reference to an external 512-byte buffer called "buff", which I couldn't find. The comment above it says:

Code: Select all

/* This code has a rather big sector buffer of 512 bytes  */
/* with this export, other code can use it, too           */


(I couldn't search for it on github because apparently github doesn't search old commits and "buff" isn't found anywhere in current commits.) If this buffer is somehow used by game code, then the bootloader would clobber game data if it writes anything to it. I wasn't clear on what was going on with buff.

These are just a few problems that indicate that this might not work. However, I thought it was an interesting idea. In addition, even if it isn't possible to use directly, could it be possible to refactor the bootloader to expose a function that could be used to read arbitrary bytes from a file at no additional time cost for flashing, without exceeding the space constraints of the bootloader? This would allow games to read data files from the SD card for "free," since it would require almost no additional program code to do this. Could this affect backwards compatibility by changing the addresses of the other functions that are currently called by the Gamebuino library? I suppose it depends on the order of all the functions.

By the way, where is the official location of the bootloader now? I dug back through the commits of the gamebuino repo and found the commit titled "move /hardware to another repo," which deleted the bootloader folder. However, looking through the other repositories under Rodot, I couldn't find a hardware one. The wiki page is out of data too. Where has it gone?

[jonnection]

By the way, where is the official location of the bootloader now? I dug back through the commits of the gamebuino repo and found the commit titled "move /hardware to another repo," which deleted the bootloader folder. However, looking through the other repositories under Rodot, I couldn't find a hardware one. The wiki page is out of data too. Where has it gone?

It seems that nobody has a "fully working copy" of the bootloader source code at the moment - a couple of people have asked me, because the current binary in use might be the one I built. My version was a modified version, that used slower SPI mode to prevent problems with SD card reading on self-made Gamebuinos. It was not anything special, just some small modifications.

I forked the Gamebuino repo a long time ago, you can look at it: https://github.com/jonneva/Gamebuino

Unfortunately, I have not saved my work either - I do not have source files that can be compiled directly. I will look if I find them from somewhere. I also did not push my changes to my forked repo (d'oh!!!)

About your idea to use the code in the bootloader to read the SD card: I am sure it could be done. Take a look at
https://github.com/jonneva/Gamebuino/bl ... /mmc_fat.c

Perhaps you could use the functions in mmc_fat to read files by calling their entry points directly by absolute address, in the same way the file loader is initiated ? But I have no idea how the return to the main code would then work, as I have not tried returning from the bootloader code.

[Sorunome]

The 512 byte buffer does not clob up any of your ram for your game, loading a game via load_game just basically wipes the entire ram of your device. Well, not really wipe it, the bootloader just pretends it's all free to use.

As you already pointed out, the functions to read are indeed just inside the code somewhere, they are not part of the jumptable and thus you'd need a seperate version of your game for each bootloader version.

"buff" is defined in mmc_fat.c

The reason with the bootloader pretending that all ram is free and that you basically don't know where your functions are located should be reason enough to just use a different SD library, SdFat is pretty simple to use. The extra code added shouldn't be significant. In fact, because of the bootloader pretending all ram to be free you'll most likely run into tons of random issues using the bootloaders SD functions, as its ram buffer stuff would collapse with the ram you use.....and thus a lot of stuff would just be corrupted.

While in theory it would be possible to refactor the bootloader it isn't practical, you have about 8 bytes free, which would be enough to add additional function references, but you would need to change some other code so that it just doesn't assume ram is free, and thus that'd take even more space and thus the bootloader would overflow. And, your game would require the device to be at the newest bootloader. Just think of all the gamebuinos out there your game would be incompatible with

And even if you pulled it off, it wouldn't read stuff for "free" from the SD card, the sd stuff in the bootloader is extremely strapped down, you'd still need to add quite a bit of abstraction on top to be able to actually use it, so you might as well just take an external library. Also, the bootloader cannot write files and works only with FAT16 and only with the 8.3 filename format.

Also @jonnection, i'm pretty sure i have a fully working copy on my machine ^.^ You can also find the source in the fork of my repo https://github.com/Sorunome/Gamebuino/t ... o_boot/src


Long story short: since the bootloader has about 8 byte of free space you'd really want to just use an external SD library.
If you are tight on memory and only want to perform reads anyways, i designed an SD library for exactly that purpose: https://github.com/Sorunome/GB_Fat/
Feel free to ask me any questions about it ^.^

[wuuff]

When I had this idea, I thought it might be crazy, but I wanted to ask the community about how doable it was. Unfortunately, from your response, I agree that this is pretty infeasible. I wouldn't be completely deterred by the bootloader being so near the size limit, or the restrictions of the stripped down functions; if it were possible to modify the bootloader to simply read a sector from a (top-level 8-character name) file into an arbitrary buffer, I might say it might be worth investigating, but the fact that the bootloader assumes all memory is free really makes this seem more trouble than it's worth. Too many difficult modifications that probably would push the size too large. Alas...

And, your game would require the device to be at the newest bootloader. Just think of all the gamebuinos out there your game would be incompatible with

I agree that it wouldn't be practical to make a game dependent on a new feature that isn't widespread, but that's not a reason not to try to add new features. Perhaps, if this had been doable, all new gamebuinos would come out with an updated bootloader, and eventually there would be enough new ones to make using the new feature practical.

Also @jonnection, i'm pretty sure i have a fully working copy on my machine ^.^ You can also find the source in the fork of my repo https://github.com/Sorunome/Gamebuino/t ... o_boot/src

That compiles to the binary that's currently in use on current gamebuinos? That's great! Perhaps I can update the bootloader page on the wiki if we can confirm that it definitely corresponds to the most up-to-date version of the bootloader.

If you are tight on memory and only want to perform reads anyways, i designed an SD library for exactly that purpose: https://github.com/Sorunome/GB_Fat/
Feel free to ask me any questions about it ^.^

How would you compare your library with PetiteFatFs? It looks like your library requires a 512-byte buffer and PetiteFatFs does not, but that probably means your library has better performance for byte-by-byte reads. What's the compiled size of the library?

By the way, Jonnection, I've been following news about the Pokitto. I might eventually try to get one, either in the kickstarter or later. It's looking pretty cool! I noticed you released a video on Facebook with my game, Armageddon!

Thanks for both of your comments, I appreciate them a lot.

[Sorunome]

If you are tight on memory and only want to perform reads anyways, i designed an SD library for exactly that purpose: https://github.com/Sorunome/GB_Fat/
Feel free to ask me any questions about it ^.^

How would you compare your library with PetiteFatFs? It looks like your library requires a 512-byte buffer and PetiteFatFs does not, but that probably means your library has better performance for byte-by-byte reads. What's the compiled size of the library?

OFC it still requires buffers, however you have to define them externally. For initing (and iirc opening a file) you still need a full 512-byte buffer, but you can easily collapse that with the screen buffer. Afterwards you can let it to only read n bytes directly into some buffer, so e.g. only read four bytes directly into a uint32_t

[jonnection]

By the way, Jonnection, I've been following news about the Pokitto. I might eventually try to get one, either in the kickstarter or later. It's looking pretty cool! I noticed you released a video on Facebook with my game, Armageddon!
Thanks for both of your comments, I appreciate them a lot.

Thanks for the comment - sorry for not crediting you in the video comments, I simply forgot to think about it. The videos come directly from my Pokitto simulator (it outputs video and sound), and so there was no editing on that video at all.

About Armageddon: it is a surprisingly enjoyable game. It is one of the Gamebuino games that has replay value.

Not many people have had the opportunity to try it, but the original Missile Command with its trackball control is an outstanding game. I came across a Missile Command cabinet a few years back. It was part of a touring exhibition of old computer tech. I don't know if it was an emulator, but the cabinet was complete with the coin slots and trackball and it was a really, really good game with a perfect difficulty curve. I went back to it about half dozen times to try to improve my score It also has great sounds and atmosphere.



@Sorunome: yep, your repository seems to have the correct source files for the bootloader. Thanks for saving it.

[Sorunome]

[...]

@Sorunome: yep, your repository seems to have the correct source files for the bootloader. Thanks for saving it.

BTW, git saves history, so you can also find it in e.g. https://github.com/Rodot/Gamebuino/tree ... o_boot/src

[Myndale]

Bit late to the party here.....

I like your thinking but in practice it won't really work. The location of routines in memory is dictated by the linker, so while it's technically possible to call bootloader functions it will almost certainly break whenever the bootloader code is modified. It was for precisely this reason that I added the jump table at the end of Flash memory which points to the "load_game" and "write_flash_page" functions (see http://gamebuino.com/wiki/index.php?title=Bootloader for details). Any bootloader functionality exposed to applications should be done via this method to ensure future compatibility.

Now even with that in place it's still not going to work how you might think it will. The bootloader is a standalone application that runs before anything else and it assumes that it has total control over the entire system, including RAM. The SD functions you talk about require a huge (by Arduino standards) 512 byte buffer to read into. Yes, you can call those bootloader functions to do SD reads but they're going to write into that RAM buffer wherever they think the bootloader has allocated it to be, which will in turn trash all over the memory that your game is using. The load_game function can get away with it of course because it makes the assumption that it never has to return to the calling program, and that the game that gets loaded never returns to the bootloader. The SD functions could be modified to accept a pointer to where that buffer is located but that would again require redistributing a new bootloader with that functionality built in.

Finally, the bootloader SD code is really crappy. When I was writing the original bootloader code Rodot expressed to me a strong wish to try to keep the code to less than 2KB, so SD support had to be kept to the bare minimum needed for the project objectives. And even then I didn't get it exactly right, IIRC jonnection had to go in afterwards and fix issues that some cards were still experiencing. Bottom line is you can't change the bootloader code, but you have total control over application code, so it's probably best to assume that the bootloader doesn't exist at all bar the very specific functions deliberately exposed by the end-of-memory jump table.