Expose offsets in reader and writer

7
closed
ncpenke
ncpenke
Posted 5 months ago

Expose offsets in reader and writer #323

Some serialization/deserialization libraries consume the writer/reader, and in this case the amount of bytes involved is lost. Surfacing the offsets of the stable storage reader and writer as a public API, would allow stable storage to be used more effectively (for example, key, value storage of serialized blobs).

ncpenke
ncpenke
Created 5 months ago

Here's the change: https://github.com/dscvr-one/cdk-rs/commit/c85d8258287f091a404abd590812d8f94e3d20a8. I can post a PR if this proposal sounds good.

paulyoung
paulyoung
Created 5 months ago

I believe I handled this in my icfs crate (which is really now just providing Read, Write, and Seek implementations)

https://github.com/codebase-labs/icfs/tree/943fa65ec89436d3438e11f620dfcca983e37811/crates/icfs

@ncpenke it's well tested so you may prefer to use that.

I created https://github.com/codebase-labs/icfs/issues/25 a while ago for potentially upstreaming the changes to cdk-rs.

paulyoung
paulyoung
Created 5 months ago

For what it's worth, my motivation for providing a different implementation was this issue which required a single struct to implement the relevant traits for everything to work together properly.

ncpenke
ncpenke
Created 5 months ago

Thanks @paulyoung, this is a much more smaller/isolated change and we've done some testing on top of it already for our use case. As part of this issue I was also hoping to get some feedback from the Dfinity team if there were any downsides to exposing the offset in this way.

As an aside, are looking into evaluating rust-fatfs so will definitely give your icfs crate a try as part of that. Really appreciate the work you've done to unify the structs for compatibility. If you can make your crate backwards compatible by type aliasing with StableWriter and StableReader it may be an easier merge path.

lwshang
lwshang
Created 5 months ago

@ncpenke Thanks for the suggestion. Feel free to send a PR.

paulyoung
paulyoung
Created 5 months ago

As an aside, are looking into evaluating rust-fatfs so will definitely give your icfs crate a try as part of that.

@ncpenke, in case you didn’t already see, there’s an example of how to do that here: https://github.com/codebase-labs/icfs/blob/943fa65ec89436d3438e11f620dfcca983e37811/examples/fatfs

It demonstrates a posix-style API but I have an issue for potentially creating something more idiomatic for Rust: https://github.com/codebase-labs/icfs/issues/23

ncpenke
ncpenke
Created 4 months ago

@lwshang, sorry for the delayed response. I opened a PR. Please let me know if you have any feedback.

@paulyoung, thanks for the info, will cycle back with you on this.