Feature request: partial backwards compatibility

[tomhampshire]

I was thinking - it would be really nice to have some backwards compatibility between schemas, especially when iterating quickly on new features/comms. There's likely some ongoing or existing work/discussions regarding this (it may already exist...), but I have not been able to find a conclusive answer (docs/wire spec/etc), so I wanted to raise it here to ask.

If certain structs were frozen, it would seem that the main thing that holds back any backwards compatibility is tagged union's variant changing their discriminant during (de)serialization. Is there a way to fix this discriminant, so that I know that particular component of the comms is going to remain stable?

For example, when going from:

#[derive(Serialize, Deserialize)]
pub enum MessageRequest {
    SetDatetime(i64),
    GetDatetime,
    ToggleLed,
}

to

#[derive(Serialize, Deserialize)]
pub enum MessageRequest {
    SetDatetime(i64),
    GetDatetime,
    ToggleLed,
    Shutdown
}

It would be great to know that the existing comms would work as before, unless you used the MessageRequest::Shutdown enum (for which the error could be handled). My naive view would be, if you could guarantee the discriminant for a variant (either by ordering, or by assigning a fixed "tag"), this partial backwards compatibility would be assured.

[jamesmunns]

In general, you can add enum variants and add fields to the end of a struct, BUT the issue is that this all generally breaks down if these types are nested. If you add a field to a struct that is in the middle of another struct, everything will go wrong.

I don't believe this is fixable with postcard's "non self describing format" design choice. The wire format is specified enough that you could potentially add a layer on top of postcard that handles some kind of schema flexibility, but it isn't possible out of the box, as far as I am aware.

[art-aier]

Is it possible to provide this compatibility for non-nested simple formats? This would be really useful (undefined fields would use default values, such as Option being None and u64 being 0).

For example:
Initially, my struct is
struct A {
a: u64,
b: u64,
}
Then I add a field c:
struct A {
a: u64,
b: u64,
c: u64,
}

[max-heller]

Is it possible to provide this compatibility for non-nested simple formats? This would be really useful (undefined fields would use default values, such as Option being None and u64 being 0).

For example: Initially, my struct is struct A { a: u64, b: u64, } Then I add a field c: struct A { a: u64, b: u64, c: u64, }

struct A {
    a: u64,
    b: u64,
    #[serde(default)]
    c: u64,
}

This simple case of adding new fields (note the addition of #[serde(default)]) to a top-level struct could be made to work by changing postcard's SeqAccess deserializer to return Ok(None) when the input is exhausted even if there are more elements expected in the sequence (instead of the current behavior of erroring with DeserializeUnexpectedEnd when attempting to deserialize an element not present in the input):

postcard/source/postcard/src/de/deserializer.rs

Lines 151 to 161 in f1c89db

	fn next_element_seed<V: DeserializeSeed<'b>>(&mut self, seed: V) -> Result<Option<V::Value>> {
	if self.len > 0 {
	self.len -= 1;
	Ok(Some(DeserializeSeed::deserialize(
	seed,
	&mut *self.deserializer,
	)?))
	} else {
	Ok(None)
	}
	}

However, the deserializer is generic over a Flavor, which does not require a way to tell if the input is exhausted or not:

postcard/source/postcard/src/de/flavors.rs

Lines 80 to 127 in f1c89db

	pub trait Flavor<'de>: 'de {
	/// The remaining data of this flavor after deserializing has completed.
	///
	/// Typically, this includes the remaining buffer that was not used for
	/// deserialization, and in cases of more complex flavors, any additional
	/// information that was decoded or otherwise calculated during
	/// the deserialization process.
	type Remainder: 'de;
	/// The source of data retrieved for deserialization.
	///
	/// This is typically some sort of data buffer, or another Flavor, when
	/// chained behavior is desired
	type Source: 'de;
	/// Obtain the next byte for deserialization
	fn pop(&mut self) -> Result<u8>;
	/// Returns the number of bytes remaining in the message, if known.
	///
	/// # Implementation notes
	///
	/// It is not enforced that this number is exactly correct.
	/// A flavor may yield less or more bytes than the what is hinted at by
	/// this function.
	///
	/// `size_hint()` is primarily intended to be used for optimizations such as
	/// reserving space for deserialized items, but must not be trusted to
	/// e.g., omit bounds checks in unsafe code. An incorrect implementation of
	/// `size_hint()` should not lead to memory safety violations.
	///
	/// That said, the implementation should provide a correct estimation,
	/// because otherwise it would be a violation of the trait’s protocol.
	///
	/// The default implementation returns `None` which is correct for any flavor.
	fn size_hint(&self) -> Option<usize> {
	None
	}
	/// Attempt to take the next `ct` bytes from the serialized message
	fn try_take_n(&mut self, ct: usize) -> Result<&'de [u8]>;
	/// Complete the deserialization process.
	///
	/// This is typically called separately, after the `serde` deserialization
	/// has completed.
	fn finalize(self) -> Result<Self::Remainder>;
	}

Postcard 2.0 could add an is_empty() method to Flavor and use it to check for the exhausted input case above.

However, this would be easy to misuse with nested structs, or by forgetting to add #[serde(default)].