CHANGES

Friday, June 17, 2016
---------------------

    Upgrade to premake5

    Some problem with 'find' in clean. Already solved in my other premakes.

    Brought across clean hack from yojimbo

    Some problems including freetype headers... wtf

    Verified that cube demos run now.

    Disabled font tool.

    Generate a cubes dataset for Mark B. Reynolds (uncompressed, non-quantized...)

    Done! Sending off!

    Update copyright and licence to The Network Protocol Company, Inc. BSD 3-Clause.

    Do some more work to build warning free. Get it done.

    Convert back to release and make sure it works. For tests I need to convert from 'CORE_ASSERT' to 'check' macro

    Bug. I fucked up the stencil shadows with a change I made to fix warnings... fix it!

    Just an accidental deletion of a line. Fixed now.

    Port to windows. 

    Hoooooly shit this is so much work.

    After much pain, got everything basically *compiling*

    Network test passes.

    Protocol test fails and hangs on reliable messages.

    WTF... the actual protocol is here is broken. I need to fix this up. What a fucking disgrace.

    Soak protocol seems to be working fine though...

    No it seems to be hanging, at least for long periods while receiving chunks. Not sure why.

    The tests being so fragmented into separate source files is incredibly obnoxious under Visual Studio.

    Next step is to get a version of ODE that is pre-built for Windows (64bit)

    Definitely will want to pre-package this as building ODE is a bit of a pain in the ass under windows.

    Got this working. Good.


Monday, Sept 7th, 2015
----------------------

    Record videos for state synchronization article:

        1. State synchronization uncompressed (basic stuff)      <--- should already look pretty much perfect

        2. Video showing popping due to not quantizing state in remote view        

            ^---------- will need to construct a large stack of objects to see the popping

            ^---------- may need to drop to lower precision to emphasize the popping

        3. Video showing quantize on both sides                  <--- should look perfect again.

        4. Video showing smoothing too slow with with large jumps

        5. Video showing better smoothing with large jumps (adaptive according to distance)

    Convert videos to web formats (need to dig up commands from CHANGES file)

    Finish writing and restructuring state synchronization article. Wow that was a lot of work.


Sunday, Sept 6th, 2015
----------------------

    Final pass over state synchronization ready for publish.

    Tell patreon supporters what I'm up to.

    Fix various bugs in state synchronization demo.

    Sketch out state sync demo functions to support uncompressed and compressed

    Templatize jitter buffer, needs to support both compressed and uncompressed state updates.

    Convert all the other bits that need uncompressed state updates.

    Implement uncompressed state sync mode.

    Hook up compressed state sync mode (not quantized)

    Hook up quantized state sync mode (quantize both sides)

    Fixed at rest handling. Current setup is "good enough". Move on.

    Split apart smoothing and adaptive smoothing modes

    Implement packet loss mode (you should see pops when input changes occur with packet loss)

    Add a display at the bottom of the screen "SYNC DISABLED" in red, so the user can clearly see 
    what is going on, that synchronization is disabled immediately, and when it is removed.

    Otherwise the videos would be a bit hard too follow.

    NOTE: If you disable sync and blow cubes around, you can watch as the grey cubes touched take a while to update

    This is a good example of lazy updates of grey cubes that would benefit from "comitting" at rest state w. reliability.


Tuesday Morning, April 7th, 2015
--------------------------------

    Removed C++ bullshit from shader loading code.

    Fixed min/max


Saturday Evening, March 14th, 2015
----------------------------------

    Export data set for Ryg to crunch on. Don't give him the result to aim for though.

    While he's working on it, calculate the best result on this data set using my existing encoding in bits.

    When he starts reporting back compression bits then I hold the cards and can tell him "not even close".

    This way I can be assured he will actually push the compression as far as possible vs. stopping as
    soon as he goes past 5% (he will, he's pragmatic...)


Saturday, March 14th, 2015
--------------------------

    Published snasphots and interpolation article.

    Implement javascript to pause video when it is not currently visible

    Polish pass the snapshot compression article.

    Work up the snapshot compression article conclusion. It's currently weak. Should be firm!

    Work out what videos need to be converted for the snapshot compression.

        1. orientation compression only

        2. compressed linear velocity

        3. without linear velocity

    Get those videos named such that they fit in with the rest and convert them to mp4 and webm

    Upload those videos to the media library

    curl those videos down to my linode.

    Move those videos to the correct folder for the article locally.

    Verify all videos are up.

    Publish article.

    Publicize article on twitter.


Monday Afternoon, March 2nd
---------------------------

    Relative index

    Absolute index

    Relative position

    Relative orientation

    Final pass over slides.

    Upload final slides to google drive:    

    Slide link is: https://drive.google.com/open?id=0BwUw_ra2ndJTdnV0M3VmaWJOTVU&authuser=0


Monday Morning, March 2nd
-------------------------

    Re-record "perfectly good" state sync end result at 4096 position

    Passed over all slides and prepped slide notes.

    Sketch out difficult slides, what I still need to do:

      Absolute index

        -- diagram to show what the skip bits looks like, eg. 900 bits (125.5bytes) just to say "no cubes have changed"
        -- diagram to show num changed cubes, changed index, cube data, etc.
        -- something to show that in the header you can choose between two encodings (one bit)
        -- since each index is 10 bits, cross-over point is about 90 cubes. more than 90 cubes changed, better off using skip bits.

      Relative index

        -- list of numbers, converted to +1, +5, +2, +10, etc.
        -- information about the encoding I'm using. variable encoding, avg. 5.5 bits per-index (almost half off)
        -- you could probably optimize this encoding over the data set but this seemed pretty good.

      Relative position

        -- delta baseline data x,y,z,base_x,base_y,base_z
        -- optimize for two level encoding. small bit per-component, otherwise large bits
        -- if any of the components don't fit into the large encoding, fall back to absolute position
        -- ruby script to run a greedy search over different small/large bit encodings and sort according to least bits
        -- results: show a table of results, winner x% reduction
        -- best result: x bits per relative position, that's just 8.5 bits per-component. Pretty good.

      Relative orientation

        -- harder. different representations to choose from
        -- axis/angle (4 values)
        -- axis multiplied by angle (3 values)
        -- relative quaternion (conjugate(base_orientation) * orientation)
        -- smallest three representation directly
        -- best result: 70% of uncompressed size (relative quaternion)
        -- second best result: 81.3% of uncompressed size (delta smallest three)


Sunday Morning, March 1st, 2015
-------------------------------

    Finish state synchronization slides:

    Priority accumulator diagram sequence

    Reliability system diagram: ack, ack_bits


Saturday Evening, February 28th, 2015
-------------------------------------

    Fill in detail for bound and quantize linear velocity

    Fill in detail for bound and quantize position

    Added diagram for linear interpolation vs. hermite interpolation with velocity

    Added diagram for error reduction (smoothing)


Saturday Lunch, February 28th, 2015
-----------------------------------

    Pass over slides and work out what videos need to be recorded again

      1. Single cube video needs to be a bit shorter (done)

      2. Cube blow video should start out with blow and have smoother motion (done)

      3. Katamari roll video introduction is too long. Don't need the big spin and spray off bit (done)

      3.5 - UDP @ 25% packet loss video needs to be shorter. Just show blow and katamari roll (done)

      6. Video with linear velocity not sent, before position compression (done)

    Lockstep demo needs a fix so the cubes don't show up until first packet.

    Fixed. Added note because this is incompatible with delta.

      4. Video showing 60 snapshots per-second uncompressed is too long. It just needs to show cube blow.

      5. Video showing orientation compression only (pretty big win -- short, blow only)

      7. Video with at rest flag, skip katamari bit.

      8. End result video for delta compression is long winded. shorten, but do show both blow and katamari

      9. Short state synchronization video (looking perfect @ 250kbps). just blow? maybe a small katamari.

    Need the math and cool bits for the smallest three. this is a big win!

    Find site that converts latex -> images (hires)

    Put formulas and working out into the slides.

    Added basic delta encoding diagrams


Saturday Morning, February 28th, 2015
-------------------------------------

    Diagram pass for deterministic lockstep

    Polish pass the input/ack articles.

    Diagram pass over snapshot interpolation.

    Setup the intro slide into the bandwidth optimization.

    Restructure slides for state synchronization. Was moving way too slow in practice

    Cut down some deterministic lockstep slides because the behavior was obvious
    and it felt drawn out going through 5 cycles of redundant inputs.


Monday Evening, February 22nd, 2015
-----------------------------------

    Take the existing slides and bring their structure back into a tree diagram

    Should see exactly what is being talked about -- identify the technical sections

    see "SLIDES" text file.


Saturday Morning, February 21st, 2015
-------------------------------------

    Tuned the simulation to work better in the sync demo.

    Basically, push apart collisions in a softer way, add some bounce for push out as well (penetration resolution)

    Adds softness but also a nice springiness.

    Added config "soften_simulation" to enable this. Turned on for sync demo only.

    Turned on for singleplayer cubes demo as well. I like it.

    Re-recorded katamari demo for physics simulation section.

    Re-record the cube video

    Re-record the cubes roll video

    Re-record the cubes blow video


Friday Evening, February 20th, 2015
-----------------------------------

    Make list of all diagrams

    1. Concept of deterministic lockstep, eg. start at same
       initial state, apply same inputs -> same result.

    2. Send all unacked packets to right side, ack packets come back,

    3. Packet structure of input packet sent (sequence, ack, inputs)

    4. Diagram showing encoding relative, one bit, input not changed.

    5. Diagram showing linear interpolation position artifacts

    6. Diagram showing linear artifacts for katamari ball (shortest path)

    7. Diagram showing how packets come in, interpolation buffer, delayed and then interpolation

    8. Diagram showing how to handle packet loss by interpolating to the next snapshot

    9. Diagram of cubes struct (from old)

    10. Diagram showing smallest three compression --> show uncompressed bits to compressed (x%)

    11. Bound and quantize linear velocity --> diagram showing uncompressed to compressed (x%)

    12. At rest flag --> diagram showing two cases, at rest, not at rest. (bits)

    13. Compressed position, bound and quantize. --> diagram showing reduction (x%)

    14. Delta compression, acks flow of packets.

    15. Buffering of snapshots on send and receive for delta encoding.

    16. Diagram of packet structure with not changed flag

    17. Diagram of packet structure with index.

    18. Diagram showing header with one bit selecting encoding.

    19. Diagram showing relative index compression, eg. two level encoding with bits, fallback to absolute

    20. Perhaps some statistics, tables or something to back up the analysis

    21. Relative position. tables or something. two level encoding, draw it.

    22. Relative orientation. diagram of two level encoding per-smallest three component delta, fallback to absolute

    23. Diagram showing packet structure sent across with state updates, and updates being applied to sim

    24. State update struct

    25. Priority accumulator diagram

    26. Algorithm in diagram, add priority, sort, pick top n.

    27. Packing as many in packet as will fit, skip past if not fit, small ones at end.

    28. Take n sent ones and clear to zero, need to bubble up from zero.

    29. Diagram showing snap position, calculate new error offset (same concept for orientation)

    30. Reliability system diagrams for header, sequence, ack, ack bits

    31. Diagram of delta encoding for state sync (bits)

    32. Diagram showing per-state update bit packing for delta


Friday Morning, February 20th, 2015
-----------------------------------

    Finish sketching out conclusion slides (pros/cons, relative rankings for bandwidth, latency...)

    Make a list of videos that need to be recorded for state synchronization

    1. Video showing input desync on packet loss

    2. Video showing jitter before quantize

    3. Video showing jitter fixed after quantize, but then after a lost input -> jitter on right (cascading failure)

    4. Video showing smoothing, looking good, and then recovering nicely from long second hitches and staying smooth.    

    Record the videos and insert into slides


Early Friday Morning (2-3AM)
----------------------------

    Sketched out state synchronization slides

    Fleshed out more slide/diagrams for deterministic lockstep


Thursday Evening, February 19th, 2015
-------------------------------------

    Sketched out keynote slides from outline:

    0. Introduction

    - what we are trying to do: network simulation from left -> right
    - what we are not talking about: topology: client/server, vs. p2p. anti-cheat. lag hiding (too much detail. derails...)
    - three main synchronization techniques: 1. deterministic lockstep, 2. snapshots and interpolation, 3. state synchronization
    - will also cover bandwidth compression techniques in detail using snapshots as a case study.
    - conclusion: pros and cons of each technique. how to select the right technique for your simulation
    - (this talk is at an advanced level. not introductory at all. the content of this talk is the state of the art at this point in time)

    1. The example simulation being networked (cube sim)

    - rigid body simulation.
    - one player controlled cube
    - 900 small cubes the player can interact with
    - all objects are gameplay effecting (not cosmetic)
    - different modes of interaction to demonstrate difficult networking cases, eg. coupled interactions between objects.
    - in general: rigid body simulation is non-linear and difficult to predict/extrapolate
    - standard networking techniques (extrapolation, dead reckoning) break down.
    - different techniques are required.

    2. Deterministic lockstep

    - bandwidth required is independent of the # of object in the world.
    - requires absolute determinism down to the bits. this is hard (summarize difficulty, but don't bog down)
    - virtually impossible to guarantee across different OS, different compiler or machine arch.
    - possible, with work, across intel/AMD, same compiler, same OS.
    - requires same framerate on both sides
    - send only inputs. *must* wait for input n before simulating frame n.
    - even though all inputs must be reliable, don't use TCP. show result of implantation over TCP @ 1%, 2%, 5% @ 250ms.
    - use UDP and redundantly send un-acked inputs each each packet
    - compression scheme for inputs => bandwidth result (50kbps @ 60 packets per-second, mostly packet header)
    - video of it it working at 25% packet loss and 250ms latency = perfect.

    3. Snapshots and interpolation

    - don't run the simulation on the right side
    - instead send only visual state and interpolate between snapshots for render
    - snapshots are the state of the entire world. eg. 900 position/orientation quat pairs + interacting (red) bool
    - snapshot packet is very large (big number). don't worry for now, we'll compress this shortly.
    - send across: strobing @ 10 pps. not very good.
    - buffer snapshots and delay them so that 2 consecutive packets can be lost without hitching
    - at 10 packets per-second, 350ms of additional delay required to handle 5% packet loss, +/- 2 frames of jitter @ 60fps (typical)
    - interpolate linear. OK but some artifacts
    - improve position interpolation by sending linear velocity. hermite interpolation
    - orientation interpolation with slerp good enough. nlerp has artifacts at 10pps (non-uniform angular speed = discontinuities).
    - end result: video @ 5% packet loss with snapshot. basically perfect. contrast with strobing. It's quite amazing we can get such a good result at such a low send rate + hermite interpolation.

    4. Bandwidth optimization case study: compressing snapshots

    - additional delay (350) @ 10pps is very large. too much.
    - would like to send snapshots more rapidly but bandwidth is off the charts
    - compress absolute quaternion: smallest 3
    - bound and compress linear velocity.
    - at rest optimization
    - why are we sending velocity at all? linear interpolation good enough at higher send rates.
    - bound position and quantize
    - as far as we can go with absolute compression
    - delta compression
    - changed bits
    - absolute indices, cross over point
    - relative index compression (measure bits required and fall back to changed bits)
    - relative position: two level encoding with optimization, fallback to absolute position
    - relative orientation: trickier. much larger % of quaternion space covered in delta vs. position. harder gains
    - delta axis/angle, delta relative quaternion large w, delta smallest 3 (90% delta base have same largest component)
    - conclusion: video showing several orders of magnitude improvement. 18mbps down to 256kbps on average.

    5. State synchronization

    - run simulation on both sides. 
    - send input and n state updates per-packet
    - use priority accumulator
    - fit as many cubes in packet as possible, given bandwidth target (may be dynamic)
    - quantize sim on both sides to avoid misprediction on right side
    - higher precision required for position/orientation due to quantization
    - snap state and use position/orientation error reduction for smoothing (local space)
    - good enough at this point: additional extra things that can be done to get even more efficient (delta, at rest optimization and so on)

    6. Conclusion

    - pros and cons of each technique. 
    - how to decide which technique is appropriate for your simulation
    - Q&A


Tuesday Morning, February 17th, 2015
------------------------------------

    Finish the advanced section at the end of the state synchronization article.


Monday Morning, February 16th, 2015
-----------------------------------

    Pass over and clean up the introduction. Was getting a bit bloviating...

    Polish pass over the introduction.

    Cut the networking 101 article. Weak. Not part of the talk. breaks flow.

    Now the article series perfectly matches the flow of the talk.


Sunday Afternoon and Evening, February 15th, 2015
-------------------------------------------------

    Cleaned up introduction to network physics series to conform to the talk content.

    Deleted additional article outlines that I will not have time to write.

    Articles should now map very, very well (perfectly) to talk content.

    Added conclusion page at end of networked physics article series. Empty for now.
    Will be used to summarize pros and cons of each technique, conclude the talk.
    It's essential for the GDC talk, and this article series must perfectly match
    that so *done*.

    Wrote the first draft of the bulk of the state synchronization article.

    Hit all the main implementation points, worked out what videos should be recorded.

    Sketched out conclusion article. It's going to need another pass but it's decent.


Sunday Morning, February 15th, 2015
-----------------------------------

    Implement code to reduce the position error over time.

    Implement code to reduce the orientation error over time (slerp towards identity)

    Make sure neither case have issues with denormals, esp. position.

    There appears to be some jittering on the right side in mode 5.

    Only appears after a while in the katamari cube

    What is going on?!

    (I suspect the input frame is getting out of sync somehow?)

    Either way, this is good. There needs to be something to do apply smoothing to.

    When a state update is applied, recalculate the position error.

    Relcalulate the orientation error.

    Verify that in smoothing mode the smoothing is working properly (katamari ball)

    Tune it so it looks as good as possible.

    Position error smoothing is now working.

    Orientation smoothing is now working.

    Tuned smoothing to look as good as possible. Orientation smoothing factor needs to be slower than position.

    There is still jitter on the right side.

    Suspect orientation smoothing is not working properly.

    Orientation error was not actually being applied, so no smoothing.

    Fixed.

    Now make sure the math is correct.

    Need 15 bits per-quaternion a,b,c to get the required orientation precision.

    Implemented two level tightness. It's definitely visually better.


Wednesday Evening, February 11th, 2015
--------------------------------------

    I was sending relative orientation with 10 bits [0,1023] precision for deltas.

    It should actually be encoded at 9 bits total [0,511] with 255 midpoint.

    This means I was encoding at twice the precision, and that relative orientation
    is likely to beat the smallest three delta now.

    Wow! I'm at 72% of absolute smallest three size!!!

        num relative quaternion samples: 189625

        absolute smallest three: 5499125 bits

        compression 4-7: 3998144 (72.7)
        compression 5-7: 4040001 (73.5)
        compression 3-7: 4091706 (74.4)
        compression 4-6: 4114374 (74.8)
        compression 5-6: 4120000 (74.9)
        compression 3-6: 4179849 (76.0)

    The snapshot compression article needs to be updated with this. This is a big improvement!


Wednesday Morning, February 11th, 2015
--------------------------------------

    Fix the * and *= operators between quat4f.

    Re-evaluate the relative quaternion statistics for delta encoding.

    It may be a lot better know. Previous statistics were completely thrown off by
    incorrect quaternion multiply!!!

    Seems to still be in the same ballpark: 90% of original size at best.

    My guess is that the smallest 3 is still the best, although it may
    be possible to get close to it, or even maybe beat it with a smart
    two-level encoding for the relative quaternion deltas.

    Nope. Relative encoding is promising. It should be explored. Do it!

    Explored it. The encoding is just slightly worse than the smallest 3 delta encode.

    eg. 81.6% best, vs. 80.8% best for smallest three delta.

    Very interesting!

    I wonder if axis-angle delta would provide wins over both of these?

    It probably would. Conclusion for compression article still stands, although
    a slightly adjustment needs to be made for the relative quaternion encoding.


Monday Morning, February 9th, 2015
----------------------------------

    Implement error reduction for position (smoothing)

    Implement error reduction for orientation (smoothing)

    Added position and orientation error arrays.

    Need a way to add the position/orientation error to the real state before render.

    Should be passed into render config because it needs to be per-view/sim.

    Added RenderConfigPerView struct, inside position/orientation error pointers initially nullptr.

    Make sure it position/orientation error is passed in to right simulation in mode 5 (smoothing) and is cleared to identity for now.

    Something crazy is going on -- multiplying the orientation by quat identity
    is resulting in a quat which is forced to identity.

    Maybe something is wrong with the * operator? Maybe it is a result of deriving from vec4f?

    Appears to be so because the goddamn quaternion * operator is just not defined.

    This explain why the relative quaternion was having problems during compression.

    Should probably return to that and try to understand if it can be made to work once this is fixed.


Sunday Morning, February 8th, 2015
----------------------------------

    Extend precision.

    CubeStateHighPrecision

    increase the position resolution until the quantize stops having artifacts

    eg. 1024, 2048 per-meter...

    increase the orientation bits to 10, is it noticeable? It's probably for the best.

    Seems that 1024 position and 10 bit orientation is sufficient.

    New modes: without quantization, with quantize, with smoothing

    Noticed that there is still quantize jitter on the katamari...

    Tried increasing the position precision. Nope.

    Tried separating linear/angular velocity precision and increasing. Nope.

    Added 64 bit compressed quaternion so I could increase orientation precision about 20 bits.

    Fixed it.

    Now reduce the precision for each back to the breaking point and then back up one

    Seems that 12 bits precision for orientation and 2048 for position/velocity does the trick.


Saturday Evening, February 7th, 2015
------------------------------------

    Would be very nice to have a button that when held disables packets getting through.

        ^--- X KEY

    Done. It's not super great because the popping hides a lot of the corrections.

    It seems that we do need additional precision

    Definitely for position. Probably also for orientation.

    Removing the quantize removes the jitter and popping on the left side,
    but on the right it still exists.

    To get it working properly it is necessary to synchronize at high(er) precision
    and do the quantize on both sides.

    It is probably a good idea to have a mode without the quantize as demonstration


Saturday Morning, February 7th, 2015
------------------------------------

    Implement jitter buffer

    Store the packet pointer and delete it? Or copy the data?

    The data is pretty small. Just copy. Safer.

    Hacked up StateJitterBuffer copied from interpolation buffer.

    Work out what needs to be done to implement delay jitter buffer.

    Add code so the sim doesn't run on the right until the first packet comes in.

    (Use the stopped flag)

    Check the jitter buffer if a state update is available for current time

    Working. State updates are coming through.

    Once the state update is pushed to sim, remove it so it is not pushed repeatedly.

    Set remote input to input in packet

    Sim is not deterministic on right. It should be!

    Work out why it's not (probably time related) and make it so.

    Seems to be deterministic until input kicks in.

    I believe it's because it is starting sim too early, so frame # is off.

    Fix should be to only start sim when the first frame should be available to sim.

    Yes. That fixed it!

    Lock in input only as first mode. This is important to make sure it is deterministic.

    Next, add some packet loss and show it desyncing (2nd mode)

    3rd mode synchronize state.

    4th mode, implement smoothing?

    Implement "ApplyStateUpdate" function to push to sim.

    INPUT_AND_STATE mode should be working now (staying in sync)

    It seems that linear/angvel in state is zero. Inverted at rest flag?

    It was inverted but now the velocity is totally fucked up.

    Incorrect velocity state on send?!

    Just a cut&paste bug on quantized cube state w. velocity load function.

    Fixed!

    Apply snapshot to sim is not pushing to sim but to view. This is incorrect. Fix it!

    Might need more precision now. It wasn't really quantizing the sim before.

    Don't need more precision, but the linear/ang velocity threshold for at rest needs to be increased,
    because some cubes are remaining active now.

    Increased to 0.5m sec threshold.

    Rename "stateful" or "state" to "sync". Much better.

    Actually implement serialize measure before cube packet write, so cubes
    perfectly fit into the allowed # of bytes and we hit requested bandwidth
    exactly.

    Hacked the code to loosely output 256kbps (equivalent to snapshot)


Friday Afternoon, February 6th, 2015
------------------------------------

    Implement priority bubble up, pull, push.

    Need to first get priority for all cubes this frame.

    This is a simple function that fills an array of floats:

        1. if red the cube has high priority
        2. if player cube extreme high priority
        3. otherwise default priority (1.0)

    Once this priority float is created (array float), then pass over existing struct 
    { float accum, int index } and add delta Time * priority to accum.

    Copy this struct and sort highest accum to smallest.

    Need a new struct for potential cubes to send: SendCubeInfo { int index; bool send; }

    Take the n most important cube indices and put them in an array for potentially
    being included in a packet.

    Fake out a function that in the future will serialize measure these cubes and mark 
    which ones actually should be sent. Right now just mark all to be sent.

    Construct state packet containing n most important cubes to be sent.

    Send packet from left to right containing most important n cube state
    that should be sent (according to measure)

    When this packet arrives print to make sure it's being received.

    Make sure send sequence is being sent.


Friday Morning, February 6th, 2015
----------------------------------

    Sketched out cube data with index for packet.

    Implement packet serialize fn. with per-cube index and serialize of position, orientation, lin/ang velocity.

    Implemented at rest flag.

    Each frame before simulate pull all state from simulation, quantize it, clamp position, linear velocity, angular velocity.

    This quantized state is where the cube state that fills the packet with n most important comes from.

    Implemented code to clamp the quantized state for position, linear/ang velocity bounds.

    Implement code to push the quantized snapshot back to the simulation

    Determine if precision needs to be higher with quantize.

    Hopefully not, but if it does some structural changes need to be made.

    Precision seems fine.


Thursday Evening, February 5th, 2015
------------------------------------

    Need to move on to stateful synchronization implementation.

    Add demo framework: "pm stateful".

    Get sim running on both sides

    Need a place to store remote input.

    Setup right simulation to run from remote input, even though it is currently always cleared.

    Fuck it just include Snapshot.h from state demo.

    Extend quantized cube state to include linear/angular velocity.

    QuantizedCubeStateWithVelocity struct.


Thursday Lunch, February 5th, 2015
----------------------------------

Convert videos to web format and upload

    webm command line is: 

        parallel 'ffmpeg -loglevel quiet -i {} -c:v libvpx -b:v 2M -c:a libvorbis -vf scale=1000:500 {.}.webm' ::: *.mov

    mp4 command line is:

        parallel 'ffmpeg -loglevel quiet -i {} -vf scale=1000:500 {.}.mp4' ::: *.mov

    Upload videos to gafferongames wordpress.

    Migrate videos across to linode.

    Hook videos up into the article.

    Test in safari.

    Test in firefox.


Thursday Morning, February 5th, 2015
------------------------------------

    Make a list of videos that need to be created.

    Fix shadows so I can record a video of the key points, eg. with huge bandwidth,
    post delta encode (changed), at the end with everything on.

    It may not have been GLFW because might be the OS upgrade.

    Trying the latest GLFW source anyway...

    It's still broken! What the fuck?!

    Fix it!

    It was fucking GLM! Fixed by converting shadow projection matrix to vectorial instead (matching non-shadow render)

    Record videos at high resolution:

        1. video of cubes blowing and katamari roll at maximum bandwidth -- 17.37 megabits per-second

        2. video showing variable bandwidth with "at rest flag" as you move around doing stuff

        3. video of compressed position @ 4.3 megabits

        4. video of delta encoding with changed flag

        5. video of reduced steady state: roll the cube off to the side and have it completely stationary

        6. video showing the end result @ 60 packets per-second


Wednesday Morning, February 4th, 2015
-------------------------------------

    Fix issue with simulator not giving correct amount of bandwidth.

    Should also handle different send rates now too, but it may oscillate a bit

    Clean up first draft.

    Go through the article and insert placeholder text for diagrams and videos to be created

    Adjust bandwidth estimates to include packet headers where they don't currently.


Monday Evening, February 2nd, 2015
----------------------------------

    position not changed 5% -- orientation not changed 5%

    dependent but opposite, so 10% gets a big win. eg. position and orientation in just 2 bits

    but... 90% of "changed" cubes have 2 bits added to their state for no reason

    is it a win? do the math!

    in order to win, we need to save more bits from 10% of cubes than we lose from 90%

    So for 1000 cubes, 50 have position not changed, 50 have orientation not changed. 900 have both changed.

    900 * 2 = 1800 bits overhead.

    50 * ( 26 - 2 ) = 1200 bits saved on orientation

    50 * ( 26 - 2 ) = 1200 bits saved on position

    So 2400 bits saved overall, average saving of 2400/1000 bits per-object, 2.4 bits per-object.

    This is a small but good win. I'll take it! :D

    Updated the article sketch out with more recent R&D


Monday Afternoon, February 2nd, 2015
------------------------------------

    Implement the best position strategy (5-9-5-9)

    And then extend it to support the extra compression for not small components

    Split out serialize relative position function so it's reusable.

    Split out serialize orientation function so it will be reusable when written. Dummy out for now w. absolute

    Update the optimize position ruby script to consider this and make sure 5-9-5-9 is still the best.

    Yes it's still the best.

    Best optimized position is 25.92 bits per-position

    That's 8.64 bits per-position component. That's fucking incredible!

    Lock it down and move on!

    Implement best orientation strategy (smallest 3 with 5 bits delta, eg. +/- 15), and large (+/-255)

    Extend to support extra range for not small components.

    Take the complicated bit of code to serialize small/large integer range
    and move it to a function. It's a real mess of cut and paste code right now!

    Done.

    Break out to optimize_smallest_three.rb

    Apply same optimization strategy for smallest three two-level (small/large bits)

    Verify the bit ranges are optimal.

    Gather statistics about it. eg. avg. bits per-orientation

    Lock it down and move on!    

    Best is 77.7% of absolute smallest 3 bits.

    5 bits small, 8 bits large.

    meaning: +/- 15 bits small, +/- 127 bits large.


Monday Morning, February 2nd, 2015
----------------------------------

    Try analysis on axis-angle (separate) with only small axis changes

    First, determine if its possible to get a win...

    50% of angles are unchanged. This alone reduces 4 components to 3 straight up...

    For example, send axis using 9 bits -- that's 18 bits plus 2 bits to indentify largest axis = 20 bits.

    This means that for 50% of the orientation values I can shave off ~9 bits.

    So on average a saving of 4.5 bits per-orientation. This is roughly in line with savings from smallest 3 delta.

    I don't think this is worth exploring in detail...

    Moving on to position.

    Start analysis on the position

    Generate uncompressed quaternion data set for dude who wants to write a blog 
    on compressing it with geometric algebra techniques ASAP

    Wait. Double check on the smallest 3 representation. What is the % of values that are identical to base?

    It's actually 5%. This is statistically significant. Might be worth one bit (quat not changed)?

    Read in position and base position data and print out

    Number of orientation values that are the same: 20196

    Number of position values that are the same: 22275

    So both around 5%

    This is a huge win. Separate changed bits for position and orientation are a win!!!

    Skip over compression of the axis/angle representation. It's a lot of work,
    and it's likely the geomerics guys are going to find something more kickass
    than my approach. Let them.

    Implement the one bit per-unchanged position, one bit per-unchanged quat and incorporate it
    into the existing bandwidth estimates for orientation compression.

    Verified. # of position/orientation values in data set equal to base = 0

    This has reduced the efficiency of my smallest 3 quaternion compression to 84.8%

    It's not a big reduction. Note that # of axis/angle angles that are identical is still 50% (!!!)

    Fixed *another* bug causing the angle valuse to cluster @ 0. Fucking clamping instead of abs! MORON!!!

    Finished upgrading all serialize fns to have bits for unchanged position, unchanged orientation.

    Basically the idea is to determine what is the best reasonable upper bound for all components
    before switching to absolute, and the upper bound for small per-component values to minimize
    bandwidth.

    This requires optimizing two variables, so add a class for the bandwidth estimate with input
    with # small bits, # large bits. Generate the set of all possibilities, calculate all and sort.
    Print out the top 10 x/y bits and % of original.

    Added the framework for this. Minimized the search space from ~4500 down to < 400 with smart ranges.

    Implement the position bandwidth estimate function.

    First, need to calculate absolute bandwidth for position.

    It is 1 bit + xy_bits * 2 + z_bits

    const float PositionBoundXY = 256;
    const float PositionBoundZ = 64;

    Position resolution is 512 per-meter, so:

    18 bits per xy component (signed)

    15 bits per-z component (unsigned)

    So that's only 3 bits more for xy components. Is the z bound a bit low maybe?

    Reduced position bound for z from 64 to 32.

    This means 14 bits per-z component now.

    Note that the # of bits for large, and the # of bits for small should be separate for xy and z

    These values behave differently!!!

    So it's really an optimization of 4 variables

    Done!

    Result of compression analysis shows that a variable encoding of 5-9-5-9 works best (~50% of absolute)

    Quaternion compression shows (~80 of original. Can't seem to easily do better than this. Moving on!)


Sunday Morning, February 1st, 2015
----------------------------------

    Try analysis on smallest 3

    Pretty good results. 

    90% of smallest 3 deltas share the same largest component, so it's a good delta opportunity.

    About 80% of absolute orientation with smallest 3 delta with 1 bit per-component and 5 bits per-companent (signed, eg. +/- 16 bits)

    This is a better result than relative quaternion.


Saturday Evening, January 31st, 2015
------------------------------------

    Study compression amounts now that the error on angle is fixed

    Now it seems that the most promising direction is the relative
    quaternion option with w inferred.

    (Need to remember flag for +/- w)

    Not sure about this.

    The effectiveness depends on the % of orientations that have the same largest component

    And the % of relative quaternions that have w as the largest component

    And the % of relative quaternions that have all components under some max value
    that makes it worthwhile. (Otherwise 10 bits per-component that isn't under the threshold = max 34 bytes)

    I do feel that some gains exist. I think they will be small though. No silver bullet!

    Therefore it is super important to do the analysis and determine what gives the best compression!

    Write out data files: "orientation_data.txt", "position_data.txt" etc.

    Start analysis on relative quaternion

    It seems that if there is a small delta xyz, it is guaranteed that there is a large w.

    This is a good thing! For small delta xyz we can always rely on w being large
    and being able to be inferred from the xyz components of the quaternion.

    Getting some benefit from it.


Saturday Lunch, January 31st, 2015
----------------------------------

    It's starting to look like separate axis angle is the best one.

    The reason is that the axis is as close or better than other quat
    representation, and 50% of the angles are *identical*.

    Therefore if the axis is within a reasonable amount of bits, say 6 bits (+1 for unchanged per-component) = 7 bits

    That's 21 bits per axis, and one bit for 50% of those.

    So basically, if axis is < 6 bits.

    Absolute:

        +2 bit -- largest 
        +9 bit -- a
        +9 bit -- b
        +9 bit -- c

        = 29 bits

    Best case relative (<50%)

        +1 bit -- relative rotation
        +1 bit -- angle the same
        +8 bit -- delta x axis
        +8 bit -- delta y axis
        +8 bit -- delta z axis

        = 26 bits (not great)

    Best case with unit length relative axis normal encoded:

        +1 bit -- relative rotation
        +1 bit -- angle the same
        +7 bit -- delta x axis
        +7 bit -- delta y axis

        = 16 bits (half = good!)

    Worst case with unit length relative axis normal:

        +1 bit -- relative rotation
        +1 bit -- angle is different
        +7 bit -- angle delta
        +7 bit -- delta x axis
        +7 bit -- delta y axis

        = 23 (slight win... 3 bits)

    Derivation for smallest three quaternion:

        (v,v,0,0) is maximum for quaternion given largest value v.

    Therefore:

        v^2 + v^2 = 1
        2v^2 = 1
        v^2 = 1/2
        v = 1 / sqrt(2)

    Derivation for smallest two normal:

        (v,v,0)

        therefore same, v = 1 / sqrt(2)

    This isn't relevant to *relative* encoding though, only absolute!

    Maybe if the largest axis value was identified (in the target),
    statistically speaking the smallest two values might have a smaller delta.

    To determine if this is the case I need to output the base/current axis-angle (separate) as a text file
    and then run analysis over this data.

    I could also run estimates of various compression schemes over this data
    and see how they work out, eg. different # of bits for axes, different # of bits for angle
    and so on -- basically, the important thing is the avg % of bits, and the % of time that
    the smaller compression scheme gets to be used.

    To work this out, run a bunch of different tests with anything from 1-10 bits for angle
    permuted with 1-10 bits for axis (that's only 100 combinations...), then output the average
    % of bits used per-orientation.

    Indeed the remaining "error" in the smallest three compression was that negative w
    was not being considered.

    Added another bool "quaternion_negative_w" to know when to negate the w.

    There is still another error that occurs, I suspect to do with smallest 2 breaking down
    (eg. blinking out of existence). 

    I may need to send more bits or identify the largest component post 

    Unfortunately the high bits on the angle not changing was caused by programmer error.

    I was not abs'ing before clamping so 50% of data was clamped to zero.

    :(


Saturday Morning, January 31st, 2015
------------------------------------

    Fix the problem encoding with quaternion.

    Found one issue. The quaternion values were / 0.5, should be / 2.0. 

    This explains the cluster @ 0.0 no change.

    Re-examine delta stats post this fix...

    Next, was dividing by integer instead of by float.

    Fixed but there are some precision issues from dropping one component. Probably need to ensure
    that we drop the largest component for numerical stability.


Friday Evening, January 30th, 2015
----------------------------------

    Now write a ruby script (scripts/delta.rb) to read in the data sets and calculate the output I want, eg. % under n bit buckets for each quantity.

    Post process data according to power of two buckets and print out % that is covered by that bucket

    Do some basic analysis for each type and try to work out which of the quaternion representations are the best

    So far it appears that with quaternion representation 90% of individual elements are the same

    This seems a bit strange, but seems to occur even if equal quaternions are discarded.

    It seems that it would be necessary to do this per-component, because although it is 90% that
    each component individually won't change, it is not 90% that all components have not changed.

    Design a relative encoding strategy for quaternion

    This encoding probably requires:

    9-10 bits per-component.

    Send three components.

    Infer the other one from unit length rule.

    Might be best to say, OK if delta of all components is within 8 bits or so (probably common...)
    then do relative encoding, and then per-component do the 1 bit changed, 8 bit delta.

    As long as we don't run into precision issues, this seems like the best bet. 

    Implement this quaternion encoding.


Friday Morning, January 30th, 2015
----------------------------------

    Measure position deltas in units and accumulate in static buffer. 

    Print out and capture to file.

    Do the same for smallest three a,b,c

    Take quaternion and convert to 10 bit precision and calculate deta (4 component)

    Calculate delta of axis * angle (3 component), quantized to 4096 (+/- 2*pi)

    Make sure all bounds are reasonable, eg. not too large

    Calculate the relative quaternion and calculate the delta of x,y,z from zero, and w from 1.0 (4 component)

    Make sure the axis lines up, eg. invert axis and angle if axis dot base axis is < 0.

    Calculate the angle difference and axis difference separately

    It may be that axis/angle compress better apart, especially if there is a significant subset where axis doesn't change

    Write all the files out to the output directory at once, eg. with fopen/fwrite
    so I can be guaranteed that all data is captured from exactly the same simulation
    and is directly comparable.

    Make sure that if the cube state is equal, it does not contribute to delta statistics (this is covered with not changed flag already...)    


Wednesday Evening, January 28th, 2015
-------------------------------------

    Separate out compression from delta

    This way there is not an unreasonable amount of compression modes and it's not confusing.

    Remove delta code from compression demo


Wednesday Afternoon, January 28th, 2015
---------------------------------------

    Implement function to count # of bits to send relative indices.

    Compare this with skip bits (900) and determine which of the two
    approaches to use dynamically on serialize write.

    Determine statistically (printf) what the average # of changed cubes
    can be per-packet before switching to skip bits.

    If # of changed cubes is > max, then fallback to skip bits even if indices more efficient.

    Tuned the relative index a bit. Getting 5 bits per-index avg. This is 50% improvement. Quite good.

    Split apart quantize linear velocity option and at rest into two separate modes.

    Work out how many bits are used for serializing linear velocity @ 20pps
    eg. hack up the linear quantize manually with # of bits by converting
    explicitly to integer.

    It's roughly 39 bits per-linear velocity. eg. 11 bits per-component.

    Could probably be reduced further.

    Make sure the ack delay is actually reasonable, for example, if there is no latency
    the ack should be almost immediate. Surely the encoding of the quaternion

    eg. print out encoding snapshot x relative to base y

    Well it's only 4 frames or so.

    Now in 4 frames difference, the change in position and orientation should be really small.

    It seems reasonable that we should be getting better orientation compression than we are currently.

    Also, seems that to get a proper delta encoding we should be adding 100ms of latency.

    Added 100ms latency. Is the relative encoding of orientation actually helping at all?

    Seems that it's not doing the best job. Need to bump up the relative position to 1024 from 512.


Tuesday Morning, January 26th, 2015
-----------------------------------

    Why is the interpolation buffer misbehaving at 60pps?

    What is triggering it? The 60pps?

    Note that if packet loss is disabled, it gets stuck forever!

    Does this mean that it's falling behind?

    If I disable jitter the hitch no longer occurs.

    Logic error to do with out of order snapshots specific to compression demo?

    No. If I add 5% packet loss hitches occur even with zero jitter (no out of order)

    It was an epsilon issue with interpolation end time vs. current time.

    Fixed.


Monday Afternoon, January 26th, 2015
------------------------------------

    Bring back absolute index on top of quantized cube state.

    Bring back relative index on top of quantized cube state.

    It may be more effective if I have a few less levels, eg. 
    4 bits then 10 bits. eg. optimize common case to 5 bits = halve.

    Settled on one bit +1, one bit 3 bits, one bit 6 bits, 10 bits fallback.

    Seems to shave 20-30kbps off the absolute index. NICE!

    Simplify relative position serialization by adding a function to 
    serialize quantized cube relative to base.

    This way I can call it from multiple places inside the relative index fn.

    Convert the relative position to also use the relative index serialize.

    Try relative orientation again.

    Seem to be working OK. Saving 10-10kbps.


Monday Lunch, January 26th, 2015
--------------------------------

    Hook up compressed quaternion to the quantized cube state.

    Tried to do relative orientation with this, but it needs a really high
    precision float per-component (0.0001), eg. 10 bits per-component to
    give a good result.

    Could try non-linear quantization with this, but first, I will try
    something simpler -- try to delta the smallest three representation
    directly.

    Explore if it is best to encode position/quat orientation with
    bits per-component, vs. one bit for range. Probably is!

    Tried it. Not really getting a lot of compression.

    At this point seems the best bet is to more roughly compress the orientation
    when the cubes are rotating quickly (not noticeable) and compress them fine
    when the rotation is small.

    The rough compression could be very rough, eg. 8X rougher

    Doing this tends to save 20-30kbps at the top end while blowing cubes.

    But, it seems to also add jitter to cubes while in the katamari.

    First try disabling angular drag. This should make the small cubes more predictable
    through the air, eg. they should maintain the exact same velocity.

    Can't disable linear drag. it's necessary for the correct look of the cube blow.

    Disable relative orientation code. It's not a big enough win


Monday Morning, January 26th, 2015
----------------------------------

    It still has some bugs, eg. not exactly reproducing the orientation (why?)

    Also, it seems that it might be possible to make it a bit more accurate,

    At minimum, use the bits perfectly for compressing the vector (eg. specify # of bits)

    First fix it so it reproduces the correct angle...

    Then if this is possible, try to optimize it -- perhaps non-linear, eg. more error
    for larger rotations or something like that, eg. take the logarthim or sqrt of the
    axis/angle multiplied value and then do the difference there, and on the other side
    exponentiate to return to linear space.

    If I change the non-relative quaternion to encode absolute the error goes away.

    Fairly certain the error is caused by random flipping of the base quaternion such
    that it is positive for the smallest three.

    To fix this I could perform quaternion quantize into 2-10-10-10 (32 bits)
    inside the quantized cube state, and then convert that to quaternion as needed.

    This way the base quaternion on both sides would be guaranteed to be in the same
    quaternion sphere half.

    Also, it seems that the cube disappearing initially is probably due to a zero axis?

    Yes, the singularity is fixed by avoiding the axis*angle encoding if the target 
    is a zero vector.

    Implement struct compressed_quaternion

    Ideally this struct would be aware of the # of bits per-component,
    perhaps templated, so it could avoid ANY differences in precision on
    either side.

    This would also allow me to easily try compression of the smallest three
    directly, to see if this has any benefit (it may! who knows!?)

    Added operator == and != to compress_quaternion

    Added serialize function and load/save.

    Should be ready to use!


Late Sunday Evening, January 25th, 2015
---------------------------------------

    Next try checking if axis is within some small amount of distance

    For example, axis within 0.1 distance (along any xyzw...)

    Encode this with a small # of bits (must be a tiny difference... eg. 4 bits per-component)

    Determine if this markedly improves the encoding or not...

    I'm not sure it would!

    For example, axis of 3 components, 4 bits each is 12 bits. 8 bits for angle difference

    Existing absolute encoding is 29bits...

    That gives 20 bits vs. 29 bits... not a huge win!

    I'm not even sure that 4 bits for the delta on the axis would be sufficient either.

    So I'm not really sure if there is anything I can do here...

    There has to be some way to get a relative encoding that is significantly cheaper
    than the smallest three...

    I mean if 4.4.4.4 is a valid encoding, then it is 16 bits which is roughly 1/2

    This should be the goal.

    Would it be possible to encode the smallest three? I'm not sure this would be wise
    because the encoding would not work across largest value changes, and this is 
    probably quite common (e.g several times per-rotation...).

    Seems like the best bet is 4.4.4.4 or 5.5.5.5 delta, eg: 16 or 20 bits.

    No frankly this is not great.

    An idea... at tangent of a 4D hypersphere is a hyperplane (3d plane)

    If I can determine the basis vectors for this hyperplane, then I could encode the 
    difference along this hyperplane using only 

    Ryg says that if you multiply the angle by the axis you get a 3 vector that can be delta'd
    and then returned to quaternion form.

    This seems to be the best way to go ahead.

    Should be able to encode delta for axis*base by 3 coords.

    But... this isn't really helping much. It *is* helping shave about 30kbps off... but it's not dramatic.


Sunday Evening January 25th, 2015
---------------------------------

    Implement relative orientation encoding.

    Can relative orientation be efficient?

    How to encode it?

    Axis-angle?

    Yes. Axis angle makes sense.

    Axis within 0.1 seems to cover all motion from cube blow.

    This also handles all motion inside katamari.

    Seems to be a winner here.

    Getting so little coverage for axis not changing, it seems
    wise to skip ahead to the axis threshold... only picking up
    basically stationary cubes with quaternion.

    Might be a win to say, OK. quaternion has not changed (1 bit)

    There was a bug in quaternion to axis/angle. Fixed.

    Seems that a separate position not changed vs. orientation not changed
    would be a big win -- looks like while the cubes are sitting coming
    to rest they are maybe unnecessarily sending orientation?

    Actually, fixed by just relaxing orientation is the same check (0.01f)

    Convert both current and base to axis-angle.

    If current axis dot base axis < 0, invert current axis.

    If current axis - base axis distance squared is within epsilon,

    set relative orientation to true.

    Determine angle difference between axis.

    Send as angle difference if less than some angle threshold.

    Work out with red coloring what is a good angle threshold (float)

    Determine if the constant axis covers a large % of rotating objects or not (blow)

    No. Constant axis does not cover a large range of objects.


Sunday January 25th, 2015
-------------------------

    Implement relative position

    Doesn't seem to be getting much improvement

    Is something wrong?!

    Lots is going wrong.

    The quantization from compressed floats is causing positions equality checks
    to not function, and I think it is also causing the incorrect ix,iy,iz.

    The solution here is to quantize the entire simulation before grabbing
    the snapshot, eg. have a new snapshot called "QuantizedSnapshot" which
    contains "QuantizedCubeState" in which positions are stored as integers.

    This way there can be no floating point issues when converting in/out
    of grid, and the grids are also perfectly efficient in terms of powers
    of two, eg. 4096 units per-grid or whatever.

    This is the next thing that must be done.

    What is the nearest power of two unit (per-meter) that obtains 0.001 precision for position?

    1/256 = 0.003

    1/512 = 0.0019

    1/1024 = 0.00097 (winner)

    ^---- is this much precision *really* necessary? seems excessive!

    Well, it's just 1/10th of a centimeter. eg. mm accuracy... maybe it is correct.

    Start here anyway. Position units are 1024 units per-meter.

    Implement quantized cube state and load/save functions to get in out of it.

    I think it's probably best to have separate sliding windows for quantized state

    This avoids unnecessary conversions

    Add members for quantized sliding window, sequence buffer, initial snapshot.

    Implement a function to get a quantized *snapshot* from the simulation

    Grab initial quantized snapshot from simulation.

    Extend code on send to put a quantized snapshot in as well as non-quantized.

    Update quantized position mode to work with quantized snapshot.

    Get delta not changed working with quantized snapshot

    Remove initial snapshot non-quantized. It's not used anymore!

    Skip right ahead from position grid to relative position

    This should provide all benefits. If necessary, implement multiple levels.

    eg. within 25cm, within 1m or so.

    Do this per-component or globally for xyz?

    Try global first and see how it goes.

    Seems to work pretty well.

    To verify common cases are getting picked up I clear the "interacting" flag
    if the position is not encoded relative.

    Was able to drop down to 512 units per-meter. Looking good.


Saturday January 24th, 2015
---------------------------

    Fix position grid bugs!!!

    Is it the cell delta, or just the grid cell conversions alone?

    It's the cell delta.

    Added code to serialize ix,iy,iz anyway and compare with delta.

    Seems to be an off by one thing for indices, eg. precision rounding?

    Could also simply be a logic error. If the logic was slightly incorrect
    then typically an incorrect cell would be off by one, statistically speaking.

    But I suspect it's precision moreso, because if the base sequence was wrong
    then the delta not changed stuff would not be working.

    One test would be to load save then load again. If it is a precision problem
    then the index values could change during this operation (they shouldn't).

    Tried this and it didn't blow up.

    Idea: compare snapshot sequence # on right side with snapshot data on left side.

    Maybe there are different snapshots on left and right side?

    Yes. The snapshot looked up in the sequence buffer doesn't always match the
    snapshot sent on the left side. I don't know why yet, but setting the base
    to lookup from the sliding window on the left instead works around the issue.

    I can use this workaround to finish the rest of the work, and then I can debug it


Friday January 23rd, 2015
-------------------------

    Seems to be some pops and glitches with delta compression.

    Also, the compression is not quite as good as I would expect.

    Something is wrong.

    Simplify the code and try to see what is going on before going further!

    It seems to me that the code that is wrong is the *relative index* code

    The delta encoding by itself is working fine!

    Also, relative index code adds a lot of complexity.

    Maybe save it for the very last optimization?

    Could also be position grid optimization. Separate it out and see!

    Move position grid ahead of relative/absolute index.

    It seems the bug is with the position grid!


Wednesday January 21st, 2015
----------------------------

    Design and code serialize_index_relative function

    It should only need one bit if the index is + 1

    It should have 2 bits for 2-6...

    3 bits for 7-...

    and so on.

    After a point (say >127), just serialialize the whole 10 bits.

    Now write the cubes that have changed using a variable bit encoding from
    previous id, eg. one bit if the next object is +1 and so on.

    Fixed bug with delta position by making epsilon smaller for changed position.

    Create new grid cube struct.

    Implement function to load grid cube state from cube state.

    Implement function to save grid cube state to cube state.

    Fixed some epsilon issues with grid cell clamp. =p

    Add new serialize function for grid cube state, copied from previous.

    Instead of serializing cube state directly, convert them all to grid cube state first (pre-pass)

    If grid cube is unchanged, one bit.

    Otherwise if changed, check if the indices are the same (common)

    If all indices are the same, copy indices from base on read.

    Split position into grid cells xyz and intra cell fractional bits

    Use this to reduce cost of position sync

    Extend delta encoding to have one bit if cell index has changed,
    therefore in the common case bit is 0, not changed and we only
    need to send fractional bits for the position

    Make sure cells are relatively large so indices change infrequently

    Serialize one bit if cell indices have not changed, otherwise
    serialize all indices at once (probably not worth having bool per-index)


Tuesday January 20th, 2015
--------------------------

    Add base (uint16_t) and initial (bool) to snapshot packet.

    Fill these in appropriately on packet send.

    This means I need to track whether an ack has been received yet.

    When an ack exists, base should be snapshot sliding window ack + 1

    Add function to fill snapshot from current simulation state.

    Rework existing compression demo to use this function to grab snapshot

    To make sure initial snapshot is valid may need to pump activation system after world init.

    Setup a new context pointer to the initial snapshot state.

    In compression demo grab initial state immediately after initialize
    and stash it away somewhere. Assert that it is valid!

    When calculating delta, if initial is true, encode with snapshot_base
    pointing to initial snapshot state.

    On serialize write, compare each object against base.

    If initial, base is a memset(0) cube struct.

    If object is identical (memcmp), write bit 0 indicating not changed.

    Otherwise write bit 1 and serialize rest of cube data.

    Write code on serialize read.

    If not changed, memcpy across the cube data from the base.

    Otherwise serialize the whole cube in-place.

    Delta encoding should be working now!

    Kinda working but doesn't seem to be exactly correct yet!

    Suspect some issues with uninitialized data and memcmp.

    Potentially issue with w coord for vec3.

    Also potentially issue with linear velocity (unused)

    Added == operator that only checks interacting, orientation, position.

    It's working great now!

    Optimize object index sending...

    So for each object 1 bit, that's n bits = 900 bits, or 113 bytes.

    If we have to identify one out of 900 objects, that's 10 bits per-id

    So 10 * 90 = 900, so the break even point is 90 objects.

    Less than 90 changed objects in a packet, send individual ids

    More than 90 changed objects, send one bit per-object

    Iterate across all cubes first, and detect set that have changed.

    Count the number of changed.

    If # changed is > 90 then use the one bit approach,

    otherwise serialize the indices (10 bits per)


Monday January 19th, 2015
-------------------------

    Insert snapshot to the sliding window on send.

    Should blow up after 256

    On receive ack, ack sequence - 1 (necessary, because we want to keep ack)

    Should no longer blow up

    Extend simulator to have context set, context ptr would have two parts

    for serialize write, it is the pointer to the sliding window.

    for serialize read, it is the the pointer to the sequence buffer.

    On receive snapshot, insert snapshot into sequence buffer

    It should now be possible to perform the serialization for all compressed
    packets in-place vs. copying across packet data, eg. serialize in place
    to the serialize write from the sliding window, and serialize read directly
    into the sequence buffer.

    This way the packets themselves only need to include the sequence #
    and then the actual data is serialized directly in/out of the buffers
    without any copies.

    Done. Works fine!


Sunday January 18th, 2015
-------------------------

    Setup ack packet.

    If a snapshot is received by the right side in a frame, send the most recent
    ack sequence # back in an ack packet (only one per-frame...)

    Copy this code from the lockstep demo

    Add printfs to verify ack packets are being sent back

    Sliding window needs a fast insert method that returns the pointer to entry

    Add sliding window and sequence buffer objects to compression struct.

    Bandwidth graph is still jittery. Why? We are sending 60 packets per-second
    and we don't have any packet loss or anything that should break it....

    AH. It's the acks being counted...

    Simulator needs a way to flag packets being sent as not counting towards bandwidth.

    Done. Bandwidth is steady again.


Saturday January 17th, 2015
---------------------------

    Render this using debug text showing the amount of bandwidth

    If the amount is greater than 512kbps then display as megabits, eg. 0.5 megabits and above

    Add smoothing to the bandwidth

    Maximum rotation speed appears to not be working for the cube rolling,
    getting shortest path twists with compression 5.

    Reduced maximum angular speed from 20 to 15.

    I am getting a fluctuating bandwidth reading.

    I believe this is because I'm getting at 10pps calculations with one packet
    dropping off the end.

    To fix I should calculate average bandwidth over the last n entries, where
    n is the size of the sliding window.

    I will set this size to 256.

    Nope. It's too delayed. 64 works pretty well.

    Made sure to increase to 60 packets per-second as well for compression demo.

    Works well.

    Added new compression mode for not sending linear velocity (sending at high rate)

    Center the bandwidth text at the top of the screen

    Generalize serialize compressed vector to take a vector min/max argument

    Bound the position so it doesn't go outside of xy and z bounds


Friday January 16th, 2015
-------------------------

    Added packet header bytes to simulator config. Default to 24 (IP+UDP header)

    Added new quaternion from rotation matrix code to vectorial:

    http://www.insomniacgames.com/converting-a-rotation-matrix-to-a-quaternion/

    Implement bandwidth tracking for all packets sent over simulator,
    take as input a header size, default to 24 bytes.

    Track sliding window of bandwidth sent across the simulator,
    initially just keep track of kbps average.

    There are two network simulators getting update pumps in the compression demo

    Which is the second one? It's not having any packets sent over it?!

    It's the one from the inactive game client.


Saturday January 10th, 2015
---------------------------

    Extend simulator so it actually performs packet serialization

    Clean up existing demo code that is manually serializing packets.

    Go across all unit tests and soak tests and remove manual packet serialize.

    Disable simulator packet serialization with client/server. It's not necessary,
    the packets are already being properly simulated via network interface (BSD).


Friday January 9th, 2015
------------------------

    Generalize serialize compressed quaternion down to n bits per-smallest 3

    Reduce quaternion bits down to minimum we can get away with (try 8, 7, 6...)

    Can't seem to go very far. See artifacts with 8.

    9 is fine. Don't see any artifacts.

    Had to increase position resolution to 0.001, eg. 1/10th of a centimeter.

    Bisect until it's not visible...

    Try 0.005... nope. 0.0025 = nope. 0.001 it is.

    
Thursday, January 8th, 2015
---------------------------

    Create new demo framework "pm compression" by copying across from snapshot.

    Take useful common stuff and move into Snapshot.h

    First mode, uncompressed.

    Getting some hitches at 60fps with 5% packet loss. WHY?!

    Sketch out compression modes:

        1. Uncompressed

        2. Compress orientation

        3. Compress at rest

        4. Compress linear velocity

        5. Compress position

    Convert across compress/decompress orientation to vectorial::quat4f

    Hook up snapshot serialize function that works according to compress mode.

    First compress orientation. It's the biggest (16 bytes).

    Can get it down to 32 bytes pretty easily.

    If the object is at rest, send one bit and don't send linear velocity.

    Many objects are at rest so this reduces bandwidth.

    Compress linear velocity.

    Reduce linear velocity resolution until it I see visual artifacts, then back up one

    Can basically not send velocity @ 60pps (no interpolation), so dropped back to 10pps for now. 

    Will go back up to 60pps once measurement is taken.

    Compress position


Sunday Afternoon, January 4th, 2015
-----------------------------------

    Write article (password protected)

    Send out messages to all supporters.

        Peter Lindgren
        Constantine Tenma
        Edward Kmett
        Pablo Riedemann
        Marc Hartsuiker
        Margaret Moser
        Michael Dwan
 

Sunday Morning, January 4th, 2015
---------------------------------

    Converting all videos to webm so they work in firefox.

    Doing this with this command line:

        parallel 'ffmpeg -loglevel quiet -i {} -c:v libvpx -b:v 2M -c:a libvorbis {.}.webm' ::: *.mov

    Seems to be working so well, it's taking up the entirity of my CPU!!! :D

    Upload webm videos

    Convert all articles to fall back to webm

    Verify articles have videos in firefox

    Sketch out remainder of series articles so people can see what is coming
    (with password protection tease...) :D

    Restructure Assets/Cube Videos/... so they are split
    up according to the article. This way I can easily
    create a bunch of new videos for an article and run
    a command to convert all videos from *.mov to *.mp4
    and *.webm without reconverting old videos (slow)


Late Saturday Evening, January 3rd, 2015
----------------------------------------

    Capture videos.

        1. Jitter @ 60pps without interpolation

        2. 10pps no interpolation

        3. 10pps with linear interpolation

        4. 10pps with hermite interpolation

        5. 10pps with extrapolation

    Upload the videos to the website

    wget the videos on to my linode

    Setup a the article with videos and link up to previous article password protected.

    (If you want to read this, please donate...)


Saturday Evening, January 3rd, 2015
-----------------------------------

    With playout delay at 250ms, code can get stuck, presumably
    missing two in a row.

    Basically, I think we need to detect if we are without snapshot,
    and the snapshot start sequence is older than the current ideal
    start snapshot, and that snapshot exists, that we can move forward!

    Confirmed that something is wrong. Increasing packet loss to 25%
    and it gets stuck within a few seconds.

    Track down why it's getting stuck.  

    Pretty sure because it doesn't look ahead more than n.

    So this settles it. Code needs to look for a new interpolation
    start sequence if start sequence gets too old.

    Added new sequence function: sequence_difference

    Gets the signed diffrence between two sequence numbers,
    assuming they are relatively close to each other, relative
    to the max sequence # / 2 that is.

    Using this to check when the old sequence is out of date
    and going into non-interpolation mode.

    Seems to be weirding out though... it's somewhat relient but then hits another
    condition where it blows up. Why?!

    Fixed. Was just a really close epsilon issue to do with floor. Stupid floating point.

    Seems necessary to have 3 frames in a row before hitch, otherwise hitches happen
    frequently enough at 5% to be annoying.

    Perhaps at 2% the hitch would not be a problem @ 250.

    Nope. Even at 2% you definitely need 3 frames!

    Need to add clamp to maximum angular velocity.

    We're getting rotational glitches at full speed, eg. 
    it rotates right back around (short path)

    Clamped linear velocity too. 

    Got decent bounds so the motion looks good, not damped.


Saturday Morning, January 3rd, 2015
-----------------------------------

    Don't send angular velocity for hermite (not used)

    Added #ifdef so I could easily bring this back (for example if hermite is working)

    Turn +/-2 frames jitter and 5% packet loss on for linear and hermite interpolation

    5% seems is too much. Seems that 2% is more reasonable.

    Once looking good at 2%, try increasing to 5% but don't use 5% if it is necessary
    to increase the amount of playout delay to compensate.

    Fixed so if no objects exist in object update, it's not pushed to view, otherwise
    simulation resets on missing interpolation snapshot and camera weirds out.

    Added nlerp to quaternion because it's easy.

    Get the interpolation buffer handling up to n missing samples.

    Fixed some bugs in the extrapolation. Step size and amount of extrapolation were unnecessarily linked.

    There still seems to be some position hitches with hermite.

    What the fuck.

    Is it dropped frames? Nope. can't be, left side doesn't hitch.

    I think it's caused by interpolation problems, *OR* late packets...

    I don't see it with no packet loss. Caused by packet loss?

    Verified that orientation hitches occurs with linearized hermite on quat,
    even with zero packet loss. Something else.

    Checking if jitter causes it, or packet loss causes it?

    Occurs with jitter off.

    Turning off packet loss again...

    Confirmed. Does not occur with packet loss off.

    But playout delay is so large, this should not be an issue.

    My guess is that there is something funny with alpha calculation
    in the case where it has to interpolate between samples 2 or 3 apart?

    Yes. It's the hermite not having the correct step size passed in!!!!

    FIXED!!!


Friday, January 2nd, 2015
-------------------------

    There is still some jitter in the interpolation.

    Seems to be limited to:

        a) position for linear interpolation (not 1st order continuous @ end points)

        b) orientation for hermite.

    I think a) is just what you get. The eye notices it.

    b) is minor and should be fixable with the log/exp trick.

    Try the log/exp

    Doesn't work perfectly immediately. Something must be wrong.

    Notice that the jitter is still there.

    Perhaps the jitter *is* a consequence of how the alpha is calculated

    Kinda got the ln/exp working but its behavior at fast rotation
    is much worst than the linear version.

    It's not really working.

    Try the trick of setting orientation to identity for linear to see if that is causing the jitter.

    Nope. There is still jitter on the position. This jitter is just a mismatch of velocity at the control points.

    Looking at position only for the 

    Try using slerp for the linear interpolation and see if this makes it look better.

    I think it probably will.

    Seems better.

    Maybe slerp is good enough for rotation.

    Hermite definitely looks better for position though.


Late Thursday Night, January 1st, 2015
--------------------------------------

    Convert the interpolation methods to output directly to view object update.

    Saves an extra conversion step / copy.

    Setup snapshot structure, eg. payload for interpolation sequence buffer.

    Setup sequence buffer.

    First just have a sequence # at which you want to render
    equivalent to playout buffer delay time.

    If this sequence entry doesn't exist return empty set.

    OK. This seems to be working fine.

    Next, take this sequence # and the next one. If either don't
    exist then return empty set, otherwise interpolate between.

    This is also working.

    Hook up so it chooses between hermite, interpolation, extrapolation
    functions according to mode.

    There seems to be some jitter on time.

    I don't think this is actually the fault of the interpolator,
    but is something wrong with what it is being fed.

    Same artifacts are visible for hermite and linear -- and nothing could be
    wrong with the linear interpolation, it's trivial!

    Get all the interpolation modes working flawlessly before moving forward.

    Fixed! It was a crazy cast from double to uint16_t directly that fucked it up.

    Casting to uint64_t first, and then to uint16_t worked (wrapping)


Thursday Evening, January 1st, 2015
-----------------------------------

    Copy playout delay buffer and call it SnapshotInterpolationBuffer

    Add playout_delay to snapshot mode data.

    Sketch out interface to get view objects for current time.

    Hook up interface if not in non-interpolated modes.

    Convert view to use vectorial math types.

    I'm sick of converting back and forth!!!!

    Remove velocity from view. It no longer needs it!

    Remove r,g,b from view update. It's now entirely a function of authority.

    Finish converting last parts of View.cpp to vectorial.

    Added vectorial::mat4f::scale method.

    Added quaternion initializer for vectorial::mat4f

    Seems like I also need to update cubes Render to use vectorial. Garrr.

    Finish coverting Cubes.cpp to new stuff that is vectorial instead of old math.

    Make sure everything still renders.

    Yep. Working!


Thursday Afternoon, January 1st, 2015
-------------------------------------

    Sketch out bit array class.

    Implement bit array class.

    Unit test the bit array class.

    Rename SlidingWindow<T> to NetworkBuffer<T>

    Rename RingBuffer<T> to SlidingWindow<T>

    Rework the NetworkBuffer<T> with hold/cold split.

    Rename NetworkBuffer<T> to SequenceBuffer<T>. More descriptive.

    Soak protocol hangs on large block receive. Something is wrong.

    Check failed: ( connection.GetCounter( protocol::CONNECTION_COUNTER_PACKETS_WRITTEN ) == NumAcks + 1 ), function test_connection, file ../tests/Protocol/TestConnection.cpp, line 69

    Get all tests passing again.

    Was an old sequence/valid combo in the connection sequence buffers going wrong.

    Improved the large block recieve by using a bit array for the received fragments. Saves memory.

    Could probably also save memory on the send, eg. hot/cold split between time last sent
    and the acked array.

    OK. I'm going to do that too.

    Done. Back to snapshots!


Thursday Morning, January 1st, 2015
-----------------------------------

    Interpolation buffer. It's actually going to be memory intensive, as it needs 
    900 cube states per-frame.

    Therefore it should not buffer a lot of unnecessary stuff...

    I think it only needs to hold 250ms of stuff.

    At 60fps that is 15 frames only...

    It would be nice if it naturally hot-cold split...

    eg. had a separation of the index data for snapshot (exists, sequence)
    and the actual data itself.

    It would be very slow to iterate across this sliding window
    if sequence and exists were 900 cube states apart.

    Perhaps the sliding window needs to be refactored to have this hot-cold
    split built in?

    Rework the sliding window so it has a hot-cold split between data it
    uses to maintain its state, and the data it stores.

    This way the sliding window and the real sliding window can both store
    the same data inside them. They are interchangeable.

    This means I need an array of exists bits, and an array of sequence.

    Work out what to call Sliding Window.

    OK. I'm renaming "RealSlidingWindow" to "RingBuffer". Leaving "SlidingWindow" alone.


Wednesday Evening, December 31st, 2014
--------------------------------------

    Think through more interpolation

    What's the best way to buffer packets and interpolate?

    It's basically a playout delay buffer.

    It needs to be smart enough not to allow any late snapshot in the buffer
    that would affect current interpolation, eg. once you start interpolating 
    from A -> C you can't allow B to pop up and affect that.

    Will delay buffer implementation.

    First I can design the data structures and the interpolation functions
    which operate on them, to get a better idea of how the buffering should
    be coded.

    Upgraded quat4f to derive from vec4f so I get all its operators for free.

    Also it's faster for linear operations (lerp!)

    Implement linear interpolation function for an array of cubes

    Sketch out mode data for hermite with extrapolation.

    It seems the extrapolation is based on extrapolation and interpolation
    with cubic splines, so it doesn't need 

    Would be a good video to demonstrate extrapolation doesn't work so well
    for rigid bodies (as previously mentioned...)

    If this is mentioned after hermite, we are already sending lin/ang vel
    so it makes sense, plus it is a cool video to see.

    Implement hacked up hermite interpolation function for an array of cubes

    Implement hacked up extrapolation for array of cubes.

    Did some more research. There are alternative ways to interpolate quaternions.

    Tim Sweeney suggests taking the logarithm of the quaternions, then just
    interpolating them linearly, and then taking the exponent of the result.

    He says this retains the correct angular motion for the quaternion.

    http://www.flipcode.com/archives/Vector_Math_Quaternions.shtml


Wednesday Morning, December 31st, 2014
--------------------------------------

    The authority color interpolation seems to be running at the 10fps speed.

    It needs to be decoupled so it pumps every frame, even if new
    view objects are not set on the view!

    Get extrapolation demo working. It's related.

    Tried it. Nope. We need to introduce smoothing and a bunch of other concepts.

    Save it for the next article! (stateful synchronization)

    Sketch out mode data for interpolation, eg. interpolation enum and setup.

    Add code to naive snapshot to only serialize the linear/angular velocity if required.

    eg. only if interpolation mode is hermite. otherwise, just position/orientation.


Tuesday, December 30th, 2014
----------------------------

    Standardized virtual shutdown method for all demos

    Standardize delete to call shutdown/initialize on current demo (if exists)

    Strange behavior with delete for snapshot demo. Doesn't seem to reset right side?

    Strange behavior with delete for lockstep demo. Doesn't seem to clear simulation,
    eg. the simulation starts right away.

    In both cases I think it is caused by not clearing the network simulator (old packets)

    Fixed.

    Standardize concept of mode across all demos. 

    Mode can be selected via 1,2,3,4,5 etc.

    Each demo needs to specify specify # of modes and initial mode # (default 0)

    Each demo can register descriptions for each mode.

    Each demo needs a "SetMode( int mode )" method.

    Demo manager should keep track of mode setting and check against 1,2,3,4,5 etc.
    and pass through to set the mode if the key pressed is within range for that demo.

    Get modes working for stone demo.

    Get modes working for lockstep demo.

    Make mode change reset demo.

    Convert all modes in lockstep and verify working.

    Boom. Nice!

    Setup mode data for snapshot demo.

    Rework snapshot demo so it passes mode data to reset

    Get the 60pps snapshots running with jitter.

    Get the 10pps snapshots running without interpolation.


Monday Evening, December 29th, 2014
-----------------------------------

    Need a way to access cube data per-simulation to copy it into the snapshot packet.

    Each frame grab the cube state and send a packet.

    On packet receive, compare sequence number and if more recent than previous, update recv sequence #.

    If more recent, push snapshot data to view (how?)

    Work out how to best push snapshot data to view?

    Initially I just want to push directly to the view.

    But eventually, we're going to want to buffer and perform interpolation.

    Just hammer it in initially?

    Trick seems to be to create an ObjectUpdate for each cube,

    Then pass in object updates to  void UpdateObjects( ObjectUpdate updates[], int updateCount, bool updateState = true );

    Some hitches. Was caused by packet loss and jitter. Disabled.

    Hacked this up. interacting not working quite right yet.

    Fixed. Grey is MaxPlayers authority, not MaxPlayers - 1.

    Orientation needs to be pushed.

    OK. The basics are working.


Monday Morning, December 29th, 2014
-----------------------------------

    Setup "pm snapshot" shortcut.

    Sketched out plans for snapshot R&D and article. Looking good!

    Get the simulation running on the left and an empty, non-running simulation on the right.

    Setup NumCubes static const int.

    Setup packet structure for naive snapshots. sequence #, n cubes, bool interacting, vec3 pos, quat4f orientation per-cube.

    Implement naive serialization function.

    Sketch out first set of intended modes for snapshot demo.

    Verify snapshot packets are getting across to right simulation with printf.


Friday, December 26th, 2014
---------------------------

    Published deterministic lockstep article.

    This is a good one!
    

Thursday Night, Christmas 2014
------------------------------

    Record videos of different network conditions with UDP/TCP:

        1. Deterministic lockstep (out of sync)

        2. Deterministic lockstep (in sync)

        3. TCP @ 100ms round trip with 1% packet loss and +/- 1 frame of jitter

        4. TCP @ 200ms round trip with 2% packet loss and +/- 1 frame of jitter

        5. TCP @ 200ms round trip with 5% packet loss and +/- 1 frame of jitter

        6. UDP @ 2 second round trip with 25% packet loss and +/- 1 frame of jitter

    All uploaded to website and ready to write the article.


Thursday Evening, Christmas 2014
--------------------------------

    Added code to switch in/out of TCP mode when you press F2.

    There was a bug in the TCP simulation. Latency and jitter were switched,
    making it look much worse than it really would be.

    It's not *great* at 100ms RTT, just occasional hitches every other second,
    but it's not as bad as that video. To get that bad you need around 250ms
    latency and 2-5% packet loss (perfect with UDP though).

    In fact UDP can go up to 25% packet loss and 2 seconds round trip without problems.

    Going up to 50% packet loss is possible if playout delay buffer is increased
    from 100ms to 250ms, but if the playout delay is 250ms, this hides the TCP
    effects at lower latencies.

    Recorded a video of the perfect deterministic behavior in splitscreen (done)

    Got a video of the perfect deterministic behavior.


Thursday Morning, Christmas 2014
--------------------------------

    250ms playout delay buffer handles 10% loss, +/- 2 frames of jitter.

    This is a good result.

    Implement a simulation of TCP retransmit behavior.

    This could even be coded such that it was included in the network simulator! eg. TCP mode!

    Basically, don't drop any packets, but deliver them all reliably and in-order,
    and make sure to take the RTT time and use that to delay subsequent packets
    by 3X TCP delay.

    This would be awesome!!!!

    Recorded a video of this. It's a super convincing argument against TCP! :D


Wednesday Evening, after Christmas Dinner, 2014
-----------------------------------------------

    Now actually implement the playout delay buffer.

    Shouldn't be too hard. Initially in stopped state.

    When first input comes in, goes into playing state and records current time value
    as "start time". Uses this later on to work out when a frame should come out.

    This means playout delay buffer needs a "playing" bool. 

    Also means it needs the timebase on "AddInputs"

    Add constant for maximum playout delay buffer size (1024)

    Resize the playout delay buffer to the correct size on startup.

    On first input, set playing to true.

    Implement the function that adds inputs to the playout delay buffer.

    This function needs to track most recent sequence added, and not add any sequences
    older than that.

    Simplest way seems to be just to iterate over all inputs to add and only add inputs
    that are exactly one older than previous most recent sequence.

    In order to get inputs acked properly, it's necessary to sequence - 1 for acks,
    otherwise the ack gets one ahead of itself, and breaks the reliability.

    We need a frame count, which will have added and be multiplied by 1/60 to get time
    to pop off the next 

    Implement function that compares time in GetFrames and works out how many frames
    to take off, with maximum number MaxSimFrames (4).

    Works basically, but it appears to be jitteringy. Eg. one frame, then none, then one.

    It's also not deterministic. Are frames being lost?

    Seems jittery every when no packet loss or jitter is in there. Has to be a logic error.

    Why is it jittery?

    It's definitely doing 1,1,1,2,0 type frame updates.

    For some reason it's not dequeing frames when it should.

    Reworked the whole push/pop and it's still jittery. WTF?!

    Fixed a bug where on read the previous input was not copied across to current.

    Fix off-by-one in the serialization of inputs.

    Drop back to non-delta compressed versions of inputs. Nope still non-det.

    Fixed off by one on input sequence when adding to playout delay buffer.

    It's deterministic now, but still jittery!

    Re-enabled delta encoding for inputs. Still deterministic!

    Why is it jittery?! Is it just that it needs interpolation?

    This seems unlikely. Packets should be received on next frame exactly. No variance.

    Woot. Got it working. Just needed some more brackets.

    Add some packet loss and jitter. It should still be smooth!

    Dropped to 2% packet loss and +/- 1 frame jitter. 

    Need 150ms playout delay to ensure this is smooth.

    It's smooth though.

    Perhaps a 250ms playout delay buffer would be suitable @ 5%?

    Packet loss seems a bit excessive. eg. causes jittering at higher latency. It shouldn't.

    There seems to be some jitter on playback that wasn't present on record, or at least I didn't notice it.

    Regardless, with playout delay of 1 second, and no packet loss or jitter, there is simply no reason
    for any sort of jitter on playback. There is plenty of buffer time!

    Nope. Jitter happens without playback. It's a thing.

    Why is jitter happening. It really shouldn't!

    All hitches are "not ready for frame n yet"

    I guess this means that there must be something wrong with the timing of the dequeue. Just so weird...

    It seems to not happen right away. Beat frequency?

    Found it! The playout delay was essentially disabled (needed to be subtracted from time, was added...)

    Now it records and plays back without any trouble!

    Wait. Nope. It's still broken :(

    Somehow the playout delay factors into it, but what I've done is actually *disable* the playout delay.

    Why would having the playout delay cause this, and not having it fixes it?

    Is it because playout delay is not a multiple of 1/60?

    Is this really just a way of telling me that we need interpolation?

    I think it probably is.

    Wait. Seem to have fixed it by adding 0.5 to the frame number. 

    I'm guessing it was really close to the current/previous frame, so it was a bit noisy
    as the float precision changed as time grew. 

    Now there is enough slop that this noise never has any effect.



Wednesday Evening, Christmas Eve, 2014
--------------------------------------

    Implement a *simple* playout delay buffer.

    This buffer should be a queue. 

    It then passes over these inputs and adds new ones to its own internal queue.

    It should track the current frame, eg. start at 0.

    From this current frame, it should be able to determine what time the next packet should come out.

    There should be a function to get inputs given server time.

    This function can return inputs that should be run.

    It's possible that more than one simulation update needs to be run, so it should return
    the number of frames, and inputs per-frame by filling an array of inputs.

    The cubes internal will need to be updated to accept a # of updates to perform per-simulation,
    as well as an array of inputs. Right now it can only run 0 or 1 simulation step.s

    There should be a maximum number of simulation frames per-update as well. I suggest 4 or 8.

    Render should only happen after the simulation updates, and a lot of work is in the render,
    so pulling off 4 invisible simulation updates then render should be OK.

    It should have a function that takes an array of inputs (from packet) and processes them.

    Added skeleton for this function. Hooked up to input packet processing.

    Add constant for MaxFrames. Set to 4.

    Sketch out function that returns the number of frames and inputs.

    Extend cubes internal to take num frames to simulate per-sim, and num inputs.

    Verify cubes demo and lockstep demo on left still works.

    Now grab the right sim frames to simulate from the playout delay buffer.


Wednesday Afternoon, December 24th, 2014
----------------------------------------

    Each frame write a packet to the right simulation (1)

    This packet should include the most recent sequence and all the inputs
    from the oldest sequence to the current (eg. num inputs)

    Inputs should be written from oldest to newest order because this is cache efficient.

    Update address to accept constructor with port num. Annoying otherwise.    

    We seem to have a memory trash on real sliding window insert! *OR* in the GetArray function.

    Implement unit tests for real sliding window.

    Isolated crash to sliding window get array function across wrap.

    Another problem with the "IsFull" check. Fixed. 

    All done. Ready to use.

    Added code to check if not full before adding to sliding window. Temporary until acks come back.

    Make sure to actually serialize read/write packets when they pop out of the simulator

    I want to make sure the serialization is actually done.

    This makes it "real networking" as far as I am concerned, vs. just passing 
    pointers to structures around.

    Verified that serialization is working and inputs are properly getting acked.

    On to playout delay buffer!


Wednesday Morning, December 24th, 2014
--------------------------------------

    Add a network simulator so I can do the packet delay and packet loss with little effort.

    Set the simulator to have 100ms latency, +/- 2 frames of jitter @ 60fps, and 5% packet loss.

    Pump the network simulator update inside demo update

    Implement code to dequeue packets from the simulator and printf for "processing them"

    Left should be address ::1 but with port 1000, right should be port 1001.

    This way the address can determine flow, and packets can go left <->right with same simulator.

    Now send a packet each frame from the left simulation to the right.

    Sequence should be frame number & 0xFFFF.

    Ack should be oldest sequence in input buffer. 

    Can get this from sliding window right? Nope. Sliding window isn't designed like that.

    Need to gather all inputs from sliding window in order of oldest to newest.

    You know. My use of a sliding window is wrong. The real sliding window has no holes.

    My "sliding window" is designed to have holes. It's really something different...

    Implement "RealSlidingWindow"

    This guy has sequence at head, and ack sequence at back. It has an interface
    to add an entry at the front of the sequence, and an interface to ack a specific
    sequence which removes all entries before and up to that one.

    It should have a constant time query for GetSequence and GetAck, and a way to 
    iterate across the array of entries, and copy entries to an array (the packet)

    Converted to real sliding window. Now I only need to put the struct I want in there,
    not a wrapper struct with "valid" and "sequence", eg. game::Input.


Tuesday Morning, December 23rd, 2014
------------------------------------

    Make escape reload the current demo

    Add a key to enable/disable determinism (F1)

    Need a constant. Max inputs. This is max inputs in packet,
    and also the max inputs in 

    At 60fps if I max the max inputs 256 then that means ~4.5 seconds.

    If 512 then 10 ~9 seconds. 1024, ~18 seconds.

    Seems that for this situation 256 should be plenty. Only going local.

    Need a sliding window for inputs.

    Set it to something large like 256. 

    Idea is if the client ever runs out of room to store the input then
    this is an error and the client would disconnect.

    Added "LockstepInternal" struct as to not expose protocol details
    to the game and demo system.

    Now I need a packet factory. Definitions for all packets.

    Converted all packet definitions to use the PROTOCOL_SERIALIZE_OBJECT macro. Nice.

    Bring across packet factory and all that stuff specific to lockstep demo protocol

    I need two packet types. Input Packet, Ack Packet.

    Input packet is current sequence, number of inputs included and delta encoded inputs.

    Ack packet is just a response saying the most recent number for which all inputs
    were received, eg. uint16_t ack. Nothing more.

    Implemented ack packet serialize.

    Implement inputs packet. Don't do delta encoding yet. Keep it simple.

    Fuck it implemented the delta encoding too. Easy.


Monday Evening, December 22nd, 2014
-----------------------------------

    We need something to pass into the update that describes the update to be performed,

    eg. which simulations to update. Which players are joined per-simulation.
    Whether local input should be applied to that simulation, and to which player.

    Whether to update that simulation *at all* should also be considered, because
    that particular simulation might be delayed (eg. driven by inputs...)

    eg. CubeUpdateConfig?

    There also needs to be a mapping from view to simulation, eg. -1 means, no mapping
    while 0 means, view gets output from simulation 0 and draws that.

    The view focus of each player should also be specified per-view, eg. view 2 may be
    viewing object 5. etc...

    This allows insertion such that cubes can be rendered entirely without simulation

    This is super important for the snapshot interpolation demo.

    eg. CubeRenderConfig?

    Added render mode. This way we can pick fullscreen, splitscreen, quadscreen.

    I also think it would be excellent to have an overlaid view, eg. overlay multiple
    simulations.

    I think the overlaid view probably wants different colors, eg. blue cubes
    force override for the second sim.

    Shadows not rendering correctly in splitscreen.

    My guess is that it's related to the disconnect between display width/height
    and the current render width/height with splitscreen.

    How best to solve this?!

    Fixed. Was incorrectly using global.displayWidth instead of displayWidth in shadow render.

    Simulation also appears quite non-deterministic. 

    Probably need to do that thing to avoid the randomization of solver order, eg. recompile ODE.

    Forum posts say that it is necessary to recompile with #define RANDOM_JOINT_ORDER commented out.

    This is out of date. 

    It now uses a pseudorandom ordering.

    Should be able to capture via:

        unsigned long dRandGetSeed()
        dRandSetSeed( unsigned long s )

    This is handy. Technically a custom install should not be required now.

    Should be deterministic now, but fragile.

    Verified. Yes it is determinstic! WOOHOO!!!! :D

    A better option would be before each simulation step, set the seed to the current simulation frame #

    (this would ensure same seed, for the *same frame* when frames are delayed via playout delay buffer...)

    Done!

    Render splitscreen divider. For now, just using a small white border where cubes aren't drawn.

    Shadow issues show up in splitscreen at 1200x500 res, but not at 1000x500 res. Why?

    Notice that it still happens at 1000x400, but not 1000x500. Something to do with aspect ratio?

    Was a different rounding error for projection matrix for cubes vs. shadows (int divide vs. float)

    Settled on 1200x500 resolution as looking best for splitscreen. Gives a square look to each side.


Monday Morning, December 22nd, 2014
-----------------------------------

    Configuration of world should be shared across all demos, eg. standardize it.

    Added way to reload demo. This recreates the demo structure from scratch, as well as reloads all assets.

    Cubes internal should take description for how many simulations there should be,
    and how many views there should be.

    For example, there could be 4 simulations but only one view.

    Code should be able to pipe arbitrarily from one simulation and render to a defined view.

    At the moment, what I need is either one simulation and one view, or two simulations and two views.

    BUT... soon for interpolation and snapshot, I'll need one simulation and two views, and a
    kickass interpolated/extrapolated thing to feed the view.

    So I should structure such that this is possible.

    Interface to create n simulations

    Interface to create n views

    Generalize code such that simulation and views are kept separate

    Make sure generalized code is working with existing singleplayer cubes demo before going further.

    Code seems to crash if I don't have a demo loaded, eg. black screen.

    Shouldn't it be clearing the screen to white?

    What is going on?

    Fixed bug. Clearing was not happening if demo was null.


Sunday, December 21st, 2014
---------------------------

    Move as much of the cubes render and so on *OUT* of the specific cubes demo.

    This allows me to have multiple cubes related demos, eg. "load lockstep", "load snapshots", "load stateful"

    Add lockstep demo framework by copying cubes demo

    Add premake setup, eg. "pm lockstep"


Saturday, December 20th, 2014
-----------------------------

    Capture a video of the player cube just rolling around by itself

    Final pass the physics simulation article so it flows conversationally.

    Updated gafferongames.com theme. It looks great!

    Added code to center the GLFW window.

    Added #if CAPTURE in ReplayManager.cpp to turn off video capture when it's not being used.

    In the future this should become a command line argument, eg. "-capture"


Sunday Evening, December 14th, 2014
-----------------------------------

    Write the article describing the physics simulation

    Seems that cloudflare doesn't handle the video static content. Unfortunate.

    The bandwidth is up to around 15 gigs today with only one article with videos.

    Seems I'm going to need to host the videos elsewhere.

    jfycat.com only handles up to 15 second clips... Some of my clips go up to 25-30 seconds.

    Youtube seems to compress the video and make it run at 30fps...

    The other website for video flowplayer.com also drops to 30 fps...

    I don't think video will allow my content to be hosted....

    Youtube appears to support 60fps only at 720p and 1080p (?!)

    My videos are not high enough resolution for that to kick in.

    Try creating a video with 720p res and see how that goes.

    1280x720 is the res. Nope. 30fps still.

    Seems that 60fps playback is only supported in chrome. Fuck that.

    Setup nginx on debian with linode.com. Hosting the mp4 files is working well.

    For $20 a month I have 3TB of transfer out. This should be enough to support 10GB a day.

    Somebody suggested cloudfront should be another option. This may be cheaper than linode.com

    http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/GettingStarted.html

    Setup S3 bucket and URL for videos once ready should be:

    http://d3ozx2w7hyub48.cloudfront.net/cubes_katamari.mp4

    Verified working! :D

    Try to capture a file in full 1080p.

    1920x1080

    Still didn't work.

    What is it? Oh look, my account is not veried on youtube. It's not serving 60fps for this reason.

    Fucking can you believe this... =p


Sunday Morning, December 14th, 2014
-----------------------------------

    Capture a bunch of videos:

        1. A video here the cube rolls around in a katamari (done)

        2. A video where the cube blows other cubes around (done)

        3. A video when the cube rolls around in normal simulation

    Guessing my webhost probably wont support self hosted videos so well.

    They seem to be getting jerky. Not sure if cloudflare or my site.

    Waiting to see if cloudflare is caching the videos. Should see a spike in bandwidth if so.

    Investigating jfycat.com

    They will encode videos at 60fps, but they have to be 15 seconds or less.

    I can *probably* work with this restriction.

    If not, I should be able to split up videos into multiple parts.

    This should take pressure off my website host, and probably provide a better
    video playback experience - hopefully.

    Youtube and other video sharing sites are out because they reduce to 30fps. =p

    Sticking with HTML5 for now. Seems the most flexible.


Saturday Evening, December 13th, 2014
-------------------------------------

    Actually deserialize the replay types.

    Implement code to handle each replay message type.

    Replay works!!! :D

    Don't show the FPS counter in playback.

    Bring back code to read the framebuffer and write to compressed TGA.

    It's already in the project in Util.h

    Implement capture mode "+capture" where each frame grabs framebuffer
    and writes out to an image (incrementing image-00001.tga etc...)

    Must also delete the previous capture directory before starting playback

    Work out how to get the framebuffer data and write it out to the TGA file
    each frame in "capture" mode.

    Remember how to convert the tga files to uncompressed videos. Quicktime Player 7 Pro.

    Make sure the quality is high and runs at 60fps.

    Compress the video with high framerate to something supported by HTML5.

    Embed the videos into my website via HTML5 video tags:

    http://www.w3schools.com/html/html5_video.asp

    Got a test video up. Looking good!


Saturday Morning, December 13th, 2014
-------------------------------------

    Setup so recording starts unless "+playback" is specified in the command line.

    Refactored command line processing so it can work with playback.

    Added logs to replay manager when recording and playback start.

    Implement code to write out events to a file.

    File structure should be:

        [header][message id][message bytes][message data] [etc...]

    This keeps it flexible and allows for variable length messages.

    Should I my own net stream to perform the serialization?

    Make sense. *YES* Otherwise I'm going to have to create my own shitty serializer for replays.

    On write. Create a write stream (max message size in bytes...), serialize
    write to this buffer, and then when serialization for that message is complete
    take the bytes of that message and write to file in format.

    On read, do that in reverse, eg. read in type, read in size, then read block
    of data and create a read stream on that.

    Should be pretty easy.

    Add message enum with ids per-replay message.

    Added nice macro to serialize objects.

    Added random seed replay message.

    Added command line message.

    Add key event message.

    Add char event message.

    Add update message (with delta time)

    Sketched out record message function.

    Implement code to open the replay file for binary write. fopen.

    Close the file on shutdown.

    Implement code to write out a message stream to file, eg: message type + stream

    Converted replay manager to have internal implementation, so impl details don't spill out.


Sunday Morning, December 7th, 2014
----------------------------------

    Sketched out replay manager.

    This manager will just be a simple record and playback events.

    The events will drive stuff like:

        - command line
        - random number seed
        - key and char events
        - update delta time

    Hook up all the bits that should call each record.

        RecordRandomSeed
        RecordCommandLine
        RecordKeyEvent
        RecordCharEvent
        RecordUpdate

    Use JSON or a binary format?

    Binary format is probably the easiest. This is not meant to be a file format
    that persists for a long time and can be played back later. It's dev only.


Sunday Morning, November 30th, 2014
-----------------------------------

    Continue optimization analysis:

    Simulation:

        34.9% - update physics
         2.2% - extrapolate objects (removed)
         1.7% - update objects (view)           <-- slerp. try converting to nlerp?
         0.7% - object update (view)            <-- mostly my crappy vector/quaternion classes
         0.3% - get view object update (view)

    Render:

        20.3% - render cube shadows
        12.4% - render cubes
        11.9% - get render state (view)
         8.0% - glFlush
         0.6% - view::Cubes::Cubes (default ctor -- nuke it!)
         0.4% - glClear

    Things that can easily be done:

        1. Convert update objects to use nlerp
        2. Don't clear the screen twice (done)
        3. Disable extrapolate objects (done -- but breaks, why?)

    Seems like lots of extra work is done in "GetViewObjects" 

        - lots of unused transforms
        - conversion from my math to vectorial
        - sorting by value (lots of copying)

    Removed the sort. I don't have any alpha blending now so it's not required.

    Post optimization:

        21.3%   - render cube shadows (lots of vec3 copy ctors, maybe pass by ref?)
        13.2%   - render cubes (11% wasted on GLM matrix multiplies. WTAF!?!)
        10.4%   - object manager get render state (mostly vectorial matrix multiply -- 7%)
        15.6%   - dWorldQuickStep
         9.0%   - dxQuadTreeSpace
         2.7%   - std::vector<short> (active object buckets?) -- duplicating the quadtree really

    Biggest wins should be:

        1. Pass by ref for generate silhouette
        2. Reduce amount of work getting render state from sim
        3. Reduce amount of GLM math done rendering cubes (possibly by replacing with vectorial matrix math...)

    Past this diminishing returns, and disruptive optimizations (Eg. removing activation system concept, reworking view mgr and so on...)

    Don't want to do those sort of optimizations yet.

    Another large optimization is to move the stencil shadows into the vertex shader.

    This is not too disruptive, but could be a reasonable amount of work. I have no idea how to do this with OpenGL yet.

    Passed in refs to generate silhouette. Faster but still slow. Best to move this to GPU long term.

    Reduce work getting render state ready.

    Tried but the inverse of the matrix is required for shadow silhoutte detection.

    Replace GLM math usage with vectorial in cube instance render.

    Just pass in the r,g,b,a directly per-cube. Don't use glm::vec4.

    ^--- After this much faster, cubes render is very small. 6.3%. 5% of that is the matrix multiply per-object!

    Remaining major optimizations that could be done:

        1. Move stencil shadows to vertex shader (would save 22% frame CPU)
        2. Convert away from my own vector math and quaternion classes
           over to vectorial everywhere. Guess, 10-15% CPU savings.

    Now profile in release build and see what's up.

    In release build it is all basicaly simulation cost, as it should be.

    Render is nearly free:

        4% - render shadows
        2% - render cubes
        1.5% - get render state (i think, under render cubes... so only 0.5% actual render cube cost)

    Simulation:

        24% - quickstep
        21% - quadtree collide

    This looks like pretty good news.

    At this point it is reasonable to stop optimization.

    There is a bug where the first cube added (bottom left corner)
    is always blue. It probably thinks it is a player owned cube.

    Worked around for now by setting MaxPlayers to 1.


Saturday Evening, November 29th, 2014
-------------------------------------

    Convert and simplify demo. There is no need for activation. All objects will be active all the time.

    Ideally, convert to 50x50 cubes (eg. 2500 cubes total)

    The cube demo gets a bit slow sometimes. Maybe try running the simulation async to render again?

    Seems to run fine in release build. Seeing as it is non-release build only, and I'm linking
    to pre-build ODE lib, I think that the issue is not physics simulation taking too much time
    but something that I am doing.

    My guess is that CPU time is being spent in the activation system (easy fix), or the stencil shadows.

    Profiling can determine this for sure.

    Ideally, I would have something that is fast enough to be running in debug build with no drama @ 60fps.

    So optimization is worthwhile.

    First profile and determine where the hotspots are.

    So far 50/50 split on render and simulation.

    Within render:

        20%   - render cubes (can't really be made much faster)
        16.1% - render cube shadows (could be moved to vertex shader...)
         9.7% - get render state (view)

    Within sim:

        41%   - game instance update
         3%   - update objects (view)
         2%   - extrapolate objects
         1%   - object manager update

    Within game instance:

        26%   - dxHashSpace collide (seems a lot!)
        7.3%  - quickstep (seems small relative to collision!)

    So basically. It seems that running render and sim parallel is probably a good idea.

    Secondly, it seem that the bulk of the cost is ODE hash collisions (broadphase).

    I suspect that with some tuning I can get the hash space behaving a bit better.    

    See http://www.ode.org/ode-0.5-userguide.html

    Based on profiling the smartest optimizations would be to:

        a) optimize the hash space collision detection in ODE

        b) run the simulation and rendering in parallel

    Got the hash collision down to 19% with some basic tuning.

    Switched to quadtree space. It's much faster now!!!

    Now look at remaining costs. What can be optimized away?

        21% - render cubes
        18% - render cube shadows
        11% - get render state (candidate for optimization!!!)

    Seems that cube instance default constructor is very expensive.

    Fucking GLM. Easy wins here.

    Now the render cubes is cheaper than the render cube shadows.

    Bug. If you hold arrow right and bring the console is up, the right input is held. 

    Demo needs to be able to check if it has input focus, if not, clear inputs to false.

    Fixed quickly by clearing inputs if console is active.

    Long term it would be better if the demo had a concept of whether it had input focus or not.


Thursday Morning, November 27th, 2014 (thanksgiving)
----------------------------------------------------

    Tried to upgrade to latest GLFW3 library, but it slowed everything down.

    I think it is rendering at a super-high retina resolution plus 8X AA,
    and the graphics card just can't handle the fill-rate.

    Not sure though. Reverted to old GLFW3 (brew)


Monday Morning, November 24th, 2014
-----------------------------------

    I would like the color tightness to grey to be slower,
    but I want to keep the tight color going to red the same.

    Tuned the wobble and bob a bit while hovering.

    Ready for talk at USC.


Sunday Evening, November 23rd, 2014
-----------------------------------

    Get shadows working!

    Seems that I can't use two sided shadows ext. Not supported in OpenGL 4.1?

    Dropping back to single pass code. Should be fast enough.

    Convert from generate silhoutte from shadow quads to triangles.

    There is a new "separate stencil" method for single pass shadow render.

    Create shadow vertex and fragment shader.

    Create shadow_vao and shadow_vbo.

    Add code to selec the vao and actually upload shadow vertex data (streaming)

    Verify the actual shadow triangles are rendering with debug vis

    Everything looks good!

    Create "Debug" shader for rendering shadow quad

    Setup a simple VAO/VBO for the shadow quad.

    Upload shadow mask vertex data    

    Render the shadow quad on top of the scene.

    Shadows are working!


Sunday Morning, November 23rd, 2014
-----------------------------------

    Focus on implementing cubes rendering with instancing (properly)

    This means I have to:

        a) create a VAO
        b) create a static VBO and IBO for the cube (easy -- but it's triangles now, not quads)
        c) create an instance buffer for cubes which contains per-cube data, eg. color, matrices etc.

    Sketch out in comments what needs to be done.

    Refactor the cubes shader to use vertex attributes for per-instance data, not uniforms.

    Added code to create the vao, vbo, instance buffer for cubes.

    Add code to setup vbo vertex attributes (vertex position, normal)

    Add code to setup per-instance vertex attributes (color, matrices etc.)

    Actually attempt to render the cubes!

    Setup light position as a uniform.

    Setup per-instance data before rendering.

    Upload per-instance data to the instance buffer.

    Kick off the instanced draw call!

    Stupid OpenGL is giving me an invalid operation post draw call. WTF... =p

    Was just that IBO was not bound to VAO.

    Get the cubes rendering... pass in the correct matrices and so forth.

    Tune the specular down a bit.

    Get the color rendering working.

    Get the alpha blending working.    

    Tuned the balance between specular, ambient, diffuse. 

    Got something that with specular now adds a nice touch vs. the old demo.

    Retuned the simulation now that I have a bit more CPU power.

    Found something that has better coming to rest behavior for stacks,
    as well as looking good while in the katamari ball. Nice!

    Bumped up activation distance. This should avoid the need to get the activation circle working.

    The demo is fast enough now to have a large activation distance, but unfortunately, there
    is a bug in the activation system above 15m activation distance.

    For now setting to 10m activation distance for the USC demo.


Sunday Morning, November 16th, 2014
-----------------------------------

    Now setup the begin/end and per-vertex calls for lit vertices.

    Begin call needs to:

        a) clear vertex index to zero
        b) select the shader program to use

    Per-vertex lit needs to:

        a) assert we are inside begin/end for lit
        b) assert we have room in the vertex buffer for another vertex
        c) copy the vertex to the buffer

    End call needs to:

        a) bind the vao and vbo
        b) actually load the vertex data as streaming
        c) setup the uniforms, eg. light position, matrices
        d) kick off the actual drawcall
        e) unbind everything and set vertex index back to zero

    What I really want to do for cubes is setup a static vertex and index buffer
    and use instancing.

    This is the *CORRECT* way to render these cubes.

    I can model this code from the other code in model, and it should be easy
    to convert across and get this running quickly.

    Frankly doing it the other way is more work!

    I can repurpose the ad-hoc glBegin/glEnd style rendering for low-perf debug only stuff.


Saturday Morning, November 15th, 2014
-------------------------------------

    Implement code to create the vertex buffer and VAO for lit and unlit vertex data.

    Bring across the code to setup the VAO from the model. Setup a VAO for lit and unlit.    

    Setup structs for each vertex type, eg. LitVertex, UnlitVertex etc.

    Setup some max # of vertices per-drawcall, eg. 1024

    Setup the actual vertex buffer data arrays

    Sketch out the begin/end calls inside render interface


Sunday Morning, November 9th, 2014
----------------------------------

    Move Render.cpp into CubesDemo.cpp -- this way cubes library has no rendering component.

    Create shaders for cubes demo

        1. Cube (basic lit vertex)
        2. CubeDebug (unlit vertex)
        3. CubeShadow (shadow vert version)

    Actually cut this down to just:

        1. CubeLit
        2. CubeUnlit

    Doesn't really seem that I can get the shader simpler than the unlit shader
    that just passes through the base color.

    Converted matrices to uniforms because I'm going to do one drawcall per-object for now.

    Sketched out a rendering path that makes it easy to emit dynamic lit and unlit vertices.

    eg. RenderBegin( RENDER_UNLIT ); RenderLitVertex( ... ); etc. RenderEnd( RENDER_UNLIT );

    This should make it easy to convert the old style OpenGL code.

    Moved a bunch of code from cubes demo class into internal, because it is simpler that way.

    Found a way to get the game input from key pressed/released events and implemented it.

    Should now have working left/right/up/down/space/z input.

    Fixed some weirdness with screen clearing.


Sunday Morning, November 2nd, 2014
----------------------------------

    Added tinycthread library. Looks promising. Will check it out.

    Get the cubes demo building and functional, but not rendering (yet)

    Hack up the Render.cpp so it just builds and cubes links.

    It's now basically working but needs rendering hooked up with modern OpenGL


Saturday Night, November 1st, 2014
----------------------------------

    Converted TestClientServer.cpp

    Fix profile and soak test client/server not compiling.

    Getting this error when running any client/server related stuff:

    Assert failed: ( value == magic ), function Check, file ../src/protocol/Stream.h, line 179

    Seems that I have fucked something up!

    Was just the order of packets. CONNECTION_PACKET must be 0 from now on.

    All but one tests passes now.

    Fix user context issue in client/server test.

    Was just an incorrect redefinition of CONTEXT_USER

    All tests pass.

    Get "TestClientServer" working.

    Get "SoakClientServer" working.

    Add new "cubes" demo, eg. "load cubes"

    Add stubbed out "interpolation" demo as well, eg. "load interpolation"

    Add command line parsing to pass into console, eg. "+demo cubes" should load cubes demo

    (It's hacked up not real, but the effect is the same for now...)

    If no command line args passed to console, default to starting with console open

    Implement premake rules: "pm cubes", "pm stone", "pm interpolation" so I can iterate quickly on demos.

    Alright! Ready to go!


Saturday Lunch, November 1st, 2014
----------------------------------

    Split client and server into their own library: "ClientServer"

    Use proper casing from now on for projects so they don't need to be single word. (Grow up!)

    Move client/server tests out of protocol so they can be iterated on independently

    Get "ClientServer" library building.

    Make sure all client server stuff is removed from protocol.

    protocol::MaxHostName should be in ClientServerConstants.h, not protocol.

    Got game client / game server compiling again.


Saturday Morning, November 1st, 2014
--------------------------------------------

    Convert the cube demo to run inside the new engine.

    Do this as prep for USC game networking talk

    First bring across as much of the cube demo engine as possible into "cubes" library.

    Moved protocol related tests into tests/protocol so there can be separate tests per-library.

    Tried to get UnitTest++ working for cubes tests, but it requires exceptions, which is a no-go.

    Convert cubes tests from UnitTest++ to normal C++ functions with CORE_ASSERT, CORE_ASSERT_CLOSE

    Split protocol tests into protocol tests and core tests, eg. "test_core"

    Split out test_network from test_protocol

    Implement unified tests via "pm test"

    Bring across virtual go tests (convert from UnitTest++)

    Fix sublime text build systems so ../external is an include directory.

    Convert *essential* virtual go tests only, as quite a bit of work is required 
    to convert tests to +z is up (right handed), rather than +y up (left handed).


Monday Evening, October 28th, 2014
----------------------------------

    Need to find a replacement for nv_dds.cpp that can handle HDR cubemaps (eg. float32 format)

    One option to explore is a library called soil2. Stupid name.

    Loosely built and installed this to evaluate.

    Nope. Don't like it.

    Found a better bunch of source based on some samples in the NVidia SDK. "nvImage"

    Created an "external" directory for external libraries, eg. nvImage

    Should probably put the forsyth code in there too, eg. misc

    Got nvImage compiling warning free.

    Removed nv_dds.cpp and code usage (#ifdef'd out)


Sunday Evening, October 27th, 2014
----------------------------------

    Consensus seems to be that I should convert to linear colorspace now before going any further.

    Research how to work in linear space and then convert to sRGB at the end with tonemapping.

    Also, should my font textures be sRGB or linear space? Which are they? Linear I assume?

    Study that article/GDC talk that Wolfgang recommended to understand what should be linear
    and what should be in sRGB.

    Study tonemapping.

    Hooked up simple Reinhardt tonemapping. Don't really see the point of it yet.

    Can't use the other filmic one, because it's got gamma correction built in, and I'm using sRGB framebuffer ext.

    Blocked now that the stupid nv_dds source doesn't support float texture formats. Fucking piece of shit.


Sunday Afternoon, October 26th, 2014
------------------------------------

    Implement a very simple lerp between pure non-metal, eg. diffuse with white specular highlight,
    and metal, eg. no diffuse with metal colored specular highlight.

    It seems to be working well.

    The clear thing that is missing here is that the input texture is not in a HDR format,
    so we don't get the cool specular indirect reflections on top of the shiny diffuse surface,
    eg. it just drops to phong.

    Also, roughness is not used to index into the cubemap in any way, so it has no effect.

    Also, it seems roughness should have some impact on the direct phong highlight, and it currently doesn't.
    
    (because I'm not using a microfacet BRDF model yet?)

    Also, missing fresnel component, which makes diffuse surfaces look cool by having more
    reflection at grazing angles. How is fresnel calculated and input to the shader?

    Need to study the existing PBR docs to understand this better.


Sunday Morning, October 26th, 2014
----------------------------------

    Make sure the cubemap tool does the right thing with blurring mip levels for roughness.    

    Reading the output, it appears that it does. Seems to be derived from cubegen actually.

    Tried out cmft with the supplied tga. Seems like it has some bugs/issues?

    Hook up code to bind the basecolor, specular, roughness, etc. by finding the shader position

    I want to follow the seb legard model, which has separate diffuse and specular colors.

    Study this model and try to wrap my head around it

    http://seblagarde.wordpress.com/2011/08/17/hello-world/

    Work out the inputs to the shader.

    They seem to be:

        BaseColor
        SpecularColor
        Roughness [0,1] (picks mip level for specular shader)
        Metallic [0,1]
        Gloss [0,1] -> SpecularPower[0,~8k]

    Setup the shader so the per-vertex (per instance actually) inputs above
    are transformed and passed to the fragment shader.

    Setup base color and specular color for stones. Try something silly, like "Blue"
    base color to gold (as per-demos I've seen...)

    Implement a scale from non-metal (left) to metal (right) on the x axis.

    Implement a scale from roughness 0 (front) to totally rough (back) on the y axis. (z is up?)

    Hack up with the existing model to see diffuse color, specular color, specular from cubemap.

    It seems that my phong light has the light position in the wrong space somehow.

    Direct phong illumination is inverted. Error in calculation? Seems that the world
    space coordinate for my light is correct! Fixed by avoiding "-s" in the reflection.


Saturday Evening, October 25th, 2014
------------------------------------

    It's getting annoying having data being deleted and using simlinks

    There should be a separation of "assets" (source inputs that are processed
    to become in-game data, and the in-game data itself).

    Processing of in-game data should be done directly from the data directory

    It seems that data should be a directory that lives in git, and should contain
    assets that shouldn't be cleaned when you go "pm clean".

    Then update the rest of the code to look for data under "../data" and everything is fine.

    Grab a bunch of cubemap assets to run through the cmft and stick them in "assets/cubemaps"

    Install cmft somewhere local (eg. /usr/local/bin) for now

    Manually convert the cubemaps from assets -> data form for now.

    If this becomes annoying, implement a tool or script to do it for me, eg. ruby script,
    or even a C++ tool like the fonttool that checks dependencies and avoids doing work
    if the data cubemaps are up to date with the assets. 


Tuesday Morning, October 21nd, 2014
-----------------------------------

    Get the model created in the stone demo.

    If the model changes for any reason (up/down/left/right), set a dirty flag, destroy the model and then recreate it (dumb)

    Some problems with rendering now. Stupid OpenGL -- hard to track down errors =p

    Some mixup with VAO creation between mesh/model. Fixed but still having problems.

    Another problem with not binding the IBO to the VAO. Stupid inconsistent OpenGL =p

    After this refactor, I can reuse VBO and IBO across multiple different models
    with different shaders and per-instance data setup. This is really flexible.


Monday Evening - October 20st, 2014
-----------------------------------

    Setup per-instance properties for go stone rendering:

        1. BaseColor
        2. Metallic
        3. Specular
        4. Roughness

    Can't add these to the vertex attribute yet, because they are not used by the shader.

    Need to split out the concept of "Model" from "Mesh" and make the mesh only the IBO/VBO.

    To create a model you must pass in both a shader *and* a mesh.

    For the moment models will be just one type, the stone model, but eventually
    there could be different model types with different vertex setups.

    For the moment do everything specific to the stone in the model,
    but eventually there could be multiple model types with different
    shaders and different per-instance data that needs setup in different
    ways.

    For example, some models would have texture coordinates in the mesh
    and would need to be setup to access those coordinates from there,
    while the stone will generate texture coordinates in the vertex shader.

    Added new model class.

    Brought across the code to create the VAO and the instance buffer.


*** RESPAWN 2 WEEK BREAK ENDS :( ***


Saturday Evening - October 18th, 2014
-------------------------------------

    Seems the reflection vector needs to be transformed from eye space into world space?

    Seems like I need to do the vertex position and normal in *world* space instead
    of eye space in order to do the cubemap, eg. I want the cubemap up to be up,
    no matter where the eye is.

    Is it better to do the calculation for the lighting in eye space or world space?

    Eric says world space.

    Convert lighting calculations to world space.

    I need to do the following:

        a) Light position in world space
        b) Eye position in world space
        c) Model matrix passed in to shader to transform point/normal into world coords,
           instead of ModelView

    Done. That was reasonably painless.

    Convert the rest of the simple diffuse/specular over to use world space coords.

    Seems to be some issue with specifying the LightPosition uniform. Why?    

    It's because I switched it from vec4 to vec3. Fixed.

    There seems to be a discontinuity of the normal where the bevel and sphere meets.

    Normals should not be discontinuous here. They should match!

    Check the normal generation for something obvious here.

    Nothing obvious.

    Inside the stone generation tool, try printing out the normal at the bevel point
    on the sphere vs. on the bevel and see if it differs

    Inside the stone generation tool, try printing out the normal at the bevel point
    on the sphere vs. on the bevel and see if it differs

    If the normal doesn't differ, it may be a tesselation issue, eg. we are not tesselating
    down to the bevel correctly?

    Seems the normals match, as expected from the construction of the bevel.

    It could be a second order thing, eg. although the normals match at the point,
    the gradient on either side leading up to it is discontinuous.

    Maybe the normals need some blending in/out at the transition. 


Saturday Morning
----------------

    Integrated NVIDIA dds loading public domain source code: nv_dds.cpp

    Compiles fine with a few small changes. Needs to be fixed not to use exceptions
    or C++ bullshit like iostream and string class =p

    Load the cubemaps, initially just inside the StoneDemo.

    Generate the OpenGL cubemap without errors.

    Seems to be crashing OpenGL with invalid enumerant?

    Seems the format was incorrect. Hardcoded to GL_RGB for now.

    Setup the filtering for the cubemap. min filter set to linear_mipmap_linear, max to linear.

    Get the cubemap rendering as part of the stone shader, eg. I want to see cubemap reflection

    So far the cubemap output is black. Not sure why, it seems I'm setting it up correctly for rendering.

    The actual vector for getting the cubemap should not just be the normal on the stone, it should be
    a reflection of the eye to point vector out to the environment. This way the cubemap should move
    as the stone moves in the world relative to cube at infinity.

    Debug why the cubemap is showing up black.

    It was because I was not specifying correct width/height per-mip level.

    Found an example Uffizi.dds from BGFX that I can use. This avoids buying HDRShop.

    Converted to CubeMap.dds that I can import in game.

    Some trouble importing but got something basic going. Doesn't look that great!


Friday Night
------------

    I need to get an environment map and run it through AMD CubeMapGen

    This tool creates a cubemap with mip levels that have different blurs.

    The blurs are then sampled with trilinear interpolation using the roughness 
    parameter as an index between mip levels.

    This tools runs in windows so I should just run it in my emulated Win8
    and then take the output files and put inside data so they can be loaded.

    Adding new cubemaps should be rare so this is OK.

    Got cubemap gen running and created a few cubemaps with CubeMapGen for testing.

    Implement 'pm cubemaps' using the same ln -s trick used for shaders.

    Cubemaps should now be linked across under bin/data dir and ready for loading.


Friday Morning
--------------

    Continue studying PBR documents collected.

    This one was particularly excellent as an overview:

    https://www.unrealengine.com/blog/physically-based-shading-on-mobile

    Created a file with snippets of shaders for PBR that I've found so far with links.

    Thinking about dependencies for moving to PBR...

    Seems that I need to get an environment map for specular reflections,
    and then run that through AMD's open source "CubeMapGen" tool such that
    it has blurs applied to different mipmap levels.

    Then I need to index those mip levels via the roughness parameter.

    This is how you get that cool roughness shading look out of PBR.

    Study how the metalness parameter of the PBR shaders work.

    What does it do fundamentally what does it change?

    It seems the metalness is the thing that brings in the indirect
    specalur, eg. the reflection. More metal = more reflection. loosly speaking.

    Seems like I should be able to get the diffuse and specular components
    from the PBR model such that they do the shading I want, but it's not
    going to look great until I have that cool indirect specular from the
    cubemap.

    Seems I should start with this cubemap. 

    I could then implement reflection added to the crap directional
    specular I currently have, and then show a bunch of stones with
    different roughness levels, = first step towards awesome.

    I need to determine if my color space for working is the standard RGB
    or if I need to move into linear space and then convert to sRGB at the
    end.

    How do I convert to linear color space?

    What about tone mapping and post-processing for HDR?

    Are these things related?

    Does my environment specular reflection need to be in HDR space or not?

    All my friends say that I should do the work in linear space and then
    convert it back to sRGB at the end with tonemapping. This seems definitive.


Wednesday Evening
-----------------

    Ported across the diffuse shader for the stone.

    Extended mesh instance data to include the normal matrix and modelview matrix.

    Extended the setup for the vao so that it sets up the new instance data.

    Properly pass in the values for modelview matrix and normal matrix.

    Added ambient term to the stone shader 0.25 so it doesn't go to black at the edges.

    Setup the correct transformed light vector uniform in eye space when rendering.

    Verify that the light does not change as the objects rotate.

    This matches the expected behavior, which is that the stones are rotating, not the camera.

    Get the light vector working for the single stone render. Seems to not be working.

    Seems that since the light position is in eye coordinates, the light position needs to be
    specified per-instance, eg. in MeshInstanceData

    Done. But there are still problems...

    Seems like there is some problem with the math... Yes. The normal matrix should be the transpose of the inverse,
    or for uniform scale, just the 3x3 submatrix of the model view will do.

    Removed the separate concept of normal matrix. I don't need it because I will only have uniform scaling.

    The idea that the light position is per-instance is wrong.

    It actually just needs to be transformed by the view matrix, not the model view matrix.

    Make the light position a uniform again.

    There seems to be something strange with the y coordinate for lighting.

    The y coordinate in inverted from what I expect it to be.

    eg. a lookat from (0,-5,0) -> (0,0,0) you would expect that a light placed at (0,-5,0) should be where the eye is

    Except it is on the opposite side.

    The values for x and z seem to be correct though.

    This is probably consequence of me thinking left handed when the coordinate system is actually left handed.

    Still doesn't make a lot of sense that the y coordinate should be flipped like this.

    Fucking hell. I was using model view projection for the normal instead of modelview. Dumbass.

    Got the specular model working at the per-vertex level. Looks good!

    Bump it up and get specular at the per-pixel level.


Wednesday Morning
-----------------

    Start research into physically based rendering (PBR)

    Study trip through the graphics pipeline series to loosely understand modern hardware.

    (Got up to part 7, will continue reading later -- my brain is full)


Tuesday Night
-------------

    I'm not sure if I like the different sizes for black and white stones.

    It might be best to keep them the same size for now, and when an optical
    illusion becomes apparent that white stones are bigger, tune experimentally 
    to make it this illusion go away.

    Make render stones take as input everything that determines how it works,
    eg. shader, mesh, array of MVP per-stone.

    Generalize mesh instance data into a struct.

    Standardize on single shader for stones, eg. always use instanced codepath

    Rework the stone demo to always use the instanced codepath.

    Remove code and shaders for non-instanced stone rendering.


Tuesday Afternoon
-----------------

    Implement demo concept.

    When client goes "load stone" a demo object should be created which then knows
    to create a bunch of stuff related to virtualgo, and knows how to render that stuff
    and so on.

    This keeps the main Game.cpp reasonably clean, eg. just setting up the basic stuff
    shared across the different demos, shaders, consoles, fonts, textures, meshes and so on.

    This is a good first step towards a general system that lets me have multiple different
    demos running outside the same framework, where each demo may respond to input differently
    and have different simulation update and rendering.

    Demos are meant to be very simple and orthogonal to networking, eg. they should mostly
    be clientside.

    Port across the single go stone into a demo.

    Extend the demo framework to handle keyboard events.

    Use up/down to increase/decrease go stone size. Start at 40 default.

    Use left/right to switch between black and white stones.

    Fix mesh data so it properly cleans up vao and vbo when destroyed

    Fix bug where RenderStones is not performing proper instance setup
    for different meshes. It's not static!

    For now, setup an instance buffer per-mesh created.

    This may be overkill but lets start here.

    It seems that the stone mesh *might* have the wrong winding order!

    Yes it does! Fix it!


Tuesday Morning
---------------

    Implement "connect" function with arguments (eg. server name)

    Implement "disconnect" function

    Implement command to reload_fonts

    Implement command to reload_meshes

    Implement command to reload_stones

    Implement command to reload_shaders

    Implement global reload command


Monday Morning
--------------

    Console function made static so in future I can make it a macro

    Implement a macro or some way of registering functions statically before the manager is created.

    Eg. should be trivial to just define:

        CONSOLE_FUNCTION( quit )
        {
            // ...
        }

    Implemented quit console function to make sure it's working.
    

Sunday Morning
--------------

    Implement function and definition to map a name to console command function pointer

    Register exit/quit function as proof of concept


Saturday Evening
----------------

    Made the cursor blink


Friday Evening
--------------

    Add the concept of current cursor position left/right within the buffer

    When inserting text with the cursor position not at end, shift the text to the right
    before inserting the new character.

    When deleting text, do it at the cursor position, delete one char to the left of cursor,
    and shift left any remaining text at or right of the cursor position left.

    Fix issue with the vertex buffers affecting each other.

    It was just a VBO being left bound, or not rebinding before putting in new data.

    Add a "_" to the font charset.

    Fix bug in the backspace mid-buffer

    Implement seek to beginning and end of buffer, eg. command-left, command-right


Friday Afternoon
----------------

    If the previous command is identical to the new command, don't add it to history. It's annoying.

    Updated "pm loc" to work again and recursively find all *.h and *.cpp files

    Add framework for rendering quads in the console cpp (eg. vao, vbo etc.)

    Implement a cursor at the text entry point.

    Seems that cursor and font vertices are interacting, much like the FPS verts were interacting.

    There is something very wrong with how I'm interfacing with OpenGL here. Need to work out what it is.

    Added console shader that doesn't have texture uv.


Friday Morning
--------------

    Add command to command history on enter, even if it doesn't match anything.

    Add command selection index, -1 means no selection.

    Add bool which is true if command history is empty.

    Implement function to find next command in buffer.

    Implement function to find previous command in buffer.

    Both can return null if the command history is empty, otherwise should always cycle.

    Implement command history, pressing UP/DOWN selects between command history in the command buffer.

    Works great.


Wednesday Evening, October 8th, 2014:
-------------------------------------

    Add input manager.

    Print out key info.

    Redirect pressed key info to input manager

    Implement mapping from key enum to chars to add to console

    (must be mod zero, *and* press or repeat action)

    Print out chars as typed.

    Create console class.

    Redirect key and char callbacks to console to handle.

    Return true from event handler if processed, false if not processed (will allow unhandled key events to pass through...)

    Implement console render method that draws text.

    NOTE: There seems to be a large discrepancy between fullscreen resolution and display res when windowed.

    I think fullscreen resolution is not retina, therefore it is faster (less fill rate)

    Maybe if I doubled the resolution requested it would give me retina fullscreen for consistency?

    Add chars to console rendered string (but only if there is room)

    There is some seriously weird shit happening with the text rendering.

    Seem to be two separate issues:

        1. Somehow the two different fonts being rendered shit on each other

        2. Something strange with the commandString values causing only the first character to be drawn (even with fps disabled)

    I don't really know what is going on here... it seems completely mysterious!

    It's something to do with the instanced rendering.

    When the instance rendering is disabled the issue stops happening.

    I think there is still a bad vertex buffer bound somewhere...

    Confirmed. If I render a normal stone without instancing, the problem doesn't occur.

    It's something happening that is triggered by instanced rendering the stones.

    What state is getting stuck on?

    For the moment, I am ignoring this so I can complete work on the console.

    Add another function for ENTER, eg. execute, which calls a callback with the current command string and clears the chars

    Implement backspace.

    Text entry seems sluggish. I think it is caused by lack of a cursor.

    Also seems less sluggish in fullscreen. I think double/triple buffering is kicking in there.

    There is still some weird interaction between the FPS display and the console text. I don't understand it. Fucking OpenGL

    It seems that the interaction occurs when you type text *before* the FPS font is first used. I think the FPS
    initialization kicks in and does something to the previous font VBO or something. FUCK ME DEAD

    Can't seem to get CTRL-C selected. It seems eaten by the OS.

    Opting to "ESCAPE" to cancel the current command, and using APPLE-Q to quit (more deliberate, but not as nice as plain ESCAPE)

    Implemented ` detection to enable/disable the console.

    Added some code to each the ` char after the console was just turned on.

    Probably need to add some code to eat the ` after the console was just disabled.

    Make sure that the console renders ON TOP OF the FPS display and everything else.

    Fixed backspace key so it respected both press and repeat actions


Wednesday Lunch, October 8th, 2014:
-----------------------------------

    Extend biconvex to get "real width" of visual shape, vs. biconvex shape width.

    Done.

    Now I need to retune the stone bevels.

    Especially the small size stones need an adjustment. Large stones seem fine.

    Only the 22 needed adjustment. The rest seem fine.


Wednesday Morning, October 8th, 2014:
-------------------------------------

    Double checked conversion to 10 bit signed. It is 100% correct.

    Something in OpenGL or the shader is quantizing the normal to 0,1

    What is it?

    Passing GL_TRUE to normalize the normal value seems to fix it.

    But why is this necessary?!

    WTF. The data is being passed in as signed 10 bit integers,
    But... when it arrives in the shader it is values in range [0,1023!]
    (eg. unsigned integer... WHY OPENGL WHY?!)

    This is dumb...

    Moving on.


Tuesday Night, October 7th, 2014:
---------------------------------

    Setup tool mesh and vertex buffer in runtime to use GL_INT_2_10_10_10_REV for normals.

    Visualize the normals as rgb color so I can see they are working.

    To convert a signed integer value, invert bits and add one (then mask off to 10 bits)

    So multiply the float by 511

    If it is less than zero, abs, then invert bits and normalize.

    Mask the value to 10 bits (for invert), and it's ready to move into place.

    For some reason the vertex normals are coming through weird.

    I don't think my conversion function is wrong. I think it's the OpenGL vbo setup?


Tuesday Afternoon, October 7th, 2014:
-------------------------------------

    Implement instanced render for go stones

    http://ogldev.atspace.co.uk/www/tutorial33/tutorial33.html

    Is it faster?

    No it's not. It seems the bottleneck is vertex processing not drawcalls.


Tuesday Morning, October 7th, 2014:
-----------------------------------

    Rendering lots of stones is slow.

    Things that can be done to optimize:

        1. Pack the vertices tighter, eg. x,y,z,nx,ny,nz instead of vec4 (done)

        2. Try just with x,y,z to see if that improves it (done -- doesn't improve it)

        3. Run the mesh optimizer over the mesh before export to .mesh (done)

        4. Reduce the tesselation level for the stone (done)

        5. Use instancing to reduce the number of draw calls (PROBABLY BIG!!!)

    Added code to detect if biconvex generate goes past 65535 indices.

    It does go past on tesselation at subdivision 6!

    Dropping to subdivision 5.

    Optimized mesh works fine. It was probably exploding because it didn't have enough bits for indices.

    To measure if optimized mesh helps, I've bumped up to 10000 meshes. This renders at 6fps (min)

    Non-optimized meshes run at 4fps.

    So mesh index order optimization is definitely helping!


Monday Night, October 6th, 2014
--------------------------------

    Added high precision time functions: core::nanoseconds and core::time.

    Render 19x19 stones without instancing, eg. one drawcall per-stone.

    We don't quite seem to be making framerate.

    Implement a framerate counter.

    Display framerate in top-right corner. Red if drops below 60 fps.

    Probably pick a different font for the framerate. Make it a bit larger.

    Make the framerate counter look cool. Red for bad framerate. Blue for good.

    Dropped to rendering only one stone.

    Framerate counter is not solid at 60FPS

    Why is this?!

    Steadied the framerate counter by accumulating time for n frames and averaging.


Monday Afternoon, October 6th, 2014:
-------------------------------------

    Optimize the various go stone meshes so they have aesthetically pleasing bevels.

    Enable multisampling AA.


Monday Morning, October 6th, 2014:
----------------------------------

    Setup function to render one stone initially.

    Will be generalized to render multiple stones via instancing later.

    Enable backface culling. Front face is CW winding order.

    Enabled depth buffer.

    Create a shader specifically for rendering the go stone derived from phong shader: Stone.frag, Stone.vert

    Work out all the uniform attributes that need to be specified

    Hacked up the uniform attributes.

    When I call glDrawElements( GL_TRIANGLES, mesh.numTriangles * 3, GL_UNSIGNED_SHORT, nullptr );

    It says invalid operation.

    Why?

    Seems to be caused by the shader.

    If I drop the shader down to something really simple, it works fine.

    Pass in proper MVP matrix for go stone.


Sunday Evening, October 5th, 2014:
----------------------------------

    I really dislike the hardcoded link between the tool side of the stones and the runtime.

    There should be no link at all, except possibly a naming convention.

    I dislike writing out to one large file for all stones.

    I would rather write out a small bit of data per-stone, eg. "Black-Size-25.stone"

    The stone manager should live in game, it should not live inside Virtual Go library.

    Bring across code from font manager that finds all '*.stone' files and parses them

    Implement code to load the stone data.

    Write something to verify that the stone manager is actually working, 

    eg. lookup a stone by name and print out its data.

    Setup new "MeshManager" around load/destroy mesh functions.

    Mesh manager should only load meshes on request, eg. don't load all meshes

    Mesh manager should also have a "Clear" method to wipe all loaded meshes.

    Implement load_mesh_data function to load in vertex and index data from the file. 

    Setup vertex buffer and load in data from file.

    Implement destroy_mesh_data to clean up the VAO.

    Setup index buffer and load in data from file.


Sunday Afternoon, October 5th, 2014:
------------------------------------

    Should virtual go continue to use vectorial, or should it convert to glm?

    I kinda prefer vectorial (simpler), less template BS.

    Keep it vectorial for now.

    WriteTGA is a useful bit of code. Should be put somewhere else though, eg. like inside game/Util.h?

    Yep. Done.

    Bring across virtual go stuff as a library, eg. "virtualgo"

    Just get it all compiling don't change to much of it yet.

    Split virtual go into tool side and runtime portions.

    Minimize the runtime portion.

    Clean up the existing mesh processing functionality w. vertex welding.

    Get the existing tool working, eg. generating a .obj file with a tool mesh.

    Research mesh optimization.

    Winner seems to be an implementation of https://home.comcast.net/~tom_forsyth/papers/fast_vert_cache_opt.html

    There are several implementations here, some of which at least are BSD so I can use them.

    http://www.martin.st/thesis/ (ZLib)
    
    https://raw.githubusercontent.com/farbrausch/fr_public/master/werkkzeug3/engine.cpp (BSD)

    https://github.com/bkaradzic/bgfx/tree/master/3rdparty/forsyth-too (Public domain)

    Skip over mesh optimization for now. It's a distraction. 

    I don't really think that I want to have the JSON hassle for generating stones.

    Instead, I know the set of stone meshes I want to generate, and that set is not going to change.

    Just hardcode the set of stones to generate.

    Implement tool to generate stone meshes.

    The generation doesn't need to run all the time.

    Instead, treat the stone meshes as static data and *check them in*

    Generate meshes initially in .obj format

    Don't generate separate black and white stones meshes yet. Leave that until later when it is visually obvious.

    Implement a stone definition struct, eg. data per-stone, parameters to generate a stone.

    Generate an array of stone definitions and then iterate across these definitions when generating stones.

    Add a stone data struct with size, mesh filename, width, height, bevel, mass, color, inertia tensor ix,iy (relevant two)

    Accumulate stone data struct as stones are generated into an array.

    Iterate across this array and write out "Stones.bin" file with header 'STONES', num stone data,
    then per-stone a binary stone data struct.

    Verify correct Stones.bin is generated.

    Symlink bin/data/stones -> data/stones as part of build process, like shaders.

    Switch to writing out in '.mesh' format (binary)


Sunday Morning, October 5th, 2014:
------------------------------------

    Clean up awful font loader code

    Font should accept an allocator and do all allocations that way.

    It should use the temp allocator to create a temporary buffer
    for the image load and conversion step.

    Remove the iostream bullshit from font loader


Friday Night, October 3rd, 2014:
--------------------------------

    Font needs a concept of Begin, DrawString, DrawString, DrawString, End.

    Switch to interleaved vertex format w. specific FontVertex struct.

    Make sure it still renders the texture atlas

    Fix ortho projection so top-left corner is (0,0)

    Convert font to pimpl, eg: FontInternal. 

    So much implementation detail is being exposed to caller.

    It's gross.

    Maybe split up font loading / font loader / builder from the font itself.

    It's annoying to have the bulk of the font file not dedicated to rendering,
    but to tedious details of how to load the fucking font file.

    Maybe split up FontRender and FontAtlas instead.

    I like this split.

    Decide if I want fonts to be 1:1 or if scaling is appropriate at runtime when rendering.

    Decide I want 1:1

    Offset the text by a small amount for the "diamond" rasterization rule
    so I can be guaranteed 1:1 mapping of texels to pixels.

    Text atlas should now render and look GREAT (hires, and with correct gamma)

    Try out a few different console fonts and pick out a good one.

    I really like the anonymous pro font at 32 point (effectively 16 point w. retina)

    Add a constant for max # of font vertices. I'm thinking 4*1024 for now.

    Assert if # verts exceeded between font begin/end. Just double it whenever it needs it.

    Add font vertex buffer inside FontRender.

    Add current vertex index inside FontRender.

    On end, pass vertex index into draw call.

    Move current code into "DrawAtlas". Verify it works using font vertex array.

    Implement proper "DrawText" with 6 verts added per char.

    Verify that test text renders properly!

    Extend DrawText to include text color as a parameter. 

    This now needs to become a vertex parameter, eg. x,y,z,r,g,b,a,u,v

    Now have text rendering with color per-strings!
    

*** RESPAWN 2 WEEK BREAK BEGINS ***


Wednesday Night, October 1st, 2014:
-----------------------------------

    Fixed the shader compilation so it prints errors out compilation properly
    and returns 0 for any shader that fails to compile. Previously was failing
    silently.

    Get basic text render showing entire texture in middle of screen

    eg. single quad with (0,0) -> (1,1) texture coords for the texture atlas.

    Verified alpha blending is working properly now. Font shader appears correct.

    For some reason the second triangle isn't rendering in the quad,
    and the texture coordinates are messed up. Not sure why :(

    urk. Missing "," in the array definition. WTF C++?

    Working now.

    Setup ortho projection for font.


Monday Night, September 29th, 2014:
--------------------------------------

    Pass over unit tests and convert them to the new library split.

    Make sure unit tests pass.

    Convert protocol soak test.

    Convert client server soak test.

    Pass over tools and make FontBuilder depend on core, eg. and move any useful functions
    defined in font builder into core so it can be reused across tools.

    Get game building again.

    *** RETURN TO PRODUCTIVE WORK ***


Sunday Night, September 28th, 2014:
-----------------------------------

    I'm thinking src/core, src/protocol and so on.

    Set up directory structure src/core, src/protocol, src/game and so on.

    Move all existing source to "src/protocol" and make sure core is built and links w. everything.

    Make sure game still compiles and links.

    Start moving stuff over to core.

    Start with "core/Common.h". Switch so it is necessary to include files across projects via prefex, eg. "core/" "protocol/" etc.

    This should include types, allocators, common, log functions, asserts, file functions (from tools)
    and so on. Core should be accessible to all libraries, tools and executables, client and server.

    Continue moving stuff until core compiles. Then conver core to "CORE_" and "namespace core".

    Split out the network related stuff from protocol into network.

    Sublime needs to be updated to include "../" as an include dir, as well as "../src"!

    Setup a way to switch between build systems easily inside Sublime, eg. "C++11 (Client)", "C++11 (Server)"

    One thing that I really dislike, the network interfaces are taking raw packets
    and instead of serializing them to packets and sending those packets, they are
    doing that internally.

    This seems wrong to me. I think the concept of an abstract packet is "protocol"
    and should be separate frome networking.

    Right now it will be too costly to fix this before returning to real work,
    for now allow network to depend on protocol and vice versa. Return to this
    and evaluate how network can be separated cleanly. I'd like network to not
    depend on protocol. Ideally, protocol would also not depend on network, but
    this may be pushing it a bit (eg. client and server clearly need network...)

    It may be possible to split out client and server, above protocol, but that seems excessive.

    Work on network until it compiles.

    core/Common.h should probably become core/Core.h? Yep.

    Work on protocol until it compiles.


Sunday Evening, September 28th, 2014:
-------------------------------------

    Need to access shader manager inside font manager in order to get font shader.

    Seems like managers need to be inside globals, eg. globals.fontManager, globals.shaderManager etc.

    I feel that I would prefer global.shaderManager instead of globals.shaderManager

    Remember, the code that I'm writing here is game code it is OK to be bound together. This is not library code!


Sunday Morning, September 28th, 2014:
-------------------------------------

    Worked through "OpenGL 4.0" ebook and added a bunch of shaders.

    I'm starting to get the hang of this. It's much simpler than old OpenGL.
    

Saturday Evening, September 27th, 2014:
---------------------------------------

    Order the ebook with company account for OpenGL 4.0 shading 
    language. Too many tutorials on the net are pre-4.0 shader lang
    so it is extremely frustrating going through old material!

    Convert triangle shader to interpolated RGB

    Get triangle shader to render rotating triangle

    glEnable( GL_DEBUG_OUTPUT ) ​to track down what is going wrong.

    Need to select the shader program before doing stuff with it. Fixed!

    Continue reading through the OpenGL 4.0 


Saturday Afternoon, September 27th, 2014:
-----------------------------------------

    Add "FontManager" class.

    Implement code to import all .font files found in bin/data/fonts/*.font

    There should be a runtime hash of all fonts, eg. lookup font by name.

    It should be possible to reload fonts at runtime.


Saturday Morning, September 27th, 2014:
---------------------------------------

    Implement shader manager than loads in all shaders in directory
    and gives them names, you can search for shaders 

    Add code to fallback to 

    Make sure my code is clean and properly cleans up all shader
    data as required on shutdown.

    Switch over existing hard coded shader load to use the shader manager instead.

    Implement function to reload shaders, this will eventually
    be called from a console command while the game is running
    whenever I edit shaders and want to see the result.

    Verify that shader reload works by randomly reloading shaders while program is running.


Tuesday Evening, September 9th, 2014:
-------------------------------------

    Research modern OpenGL style.

    Seems I have an OpenGL 4.1 on my macbook.

    Shader model is 410.

    Research how to get shaders built and working for 410.,
    the example source I am working from doesn't seem to work
    (it appears to be older style?)

    Get a basic triangle up and being rendered.

    Implement helper functions to create shaders and so on.

    Move the shaders out of the C++ and in to shaders/*

    Copy across shaders in build step. It's error prone though
    because you can accidentally edit the wrong shader file, 
    and have that clobbered.

    Implement ln -s bin/data/shaders -> data/shaders so I don't
    have to copy the shaders across all the time in the build step.

    Don't use os.rmdir for clean on "bin" directory. It follows softlinks!


Tuesday Morning, September 2nd, 2014:
-------------------------------------

    Enabled SRGB framebuffer with glEnable( GL_FRAMEBUFFER_SRGB );

    Verified this is functional by clearing to (0.5) grey, with SRGB
    framebuffer this is quite light, easily 50% grey -- without, it is
    quite dark, more like 30% dark grey!


Monday Evening, September 1st, 2014:
------------------------------------

    Converted OpenGL windowing over to GLFW

    Got GLEW working and added code to pump and remove all gl errors post
    GLEW init (it seems to get a lot of errors trying out stuff?)

    Seems I now need to do a large conversion from old style OpenGL
    to modern style OpenGL in order to get the fonts working again.


Monday Lunch, September 1st, 2014:
----------------------------------

    Track down why font texture is rendering as white quads.

    .font file seems to have valid texture data that looks like
    glyhps, so the error must be on the runtime side.

    Was missing glEnable( GL_TEXTURE ). Derp.

    Font looks really aliased around the edges. Why?

    Created higher resolution texture and then scaled down.

    Doesn't really seem to help much. Still looks bad.

    Tried a bunch of different fonts. So far only fonts that are
    bitmap fonts, eg. no AA are looking good.

    Found the issue. 

    GLUT does not support retina display, so I'm effectively rendering at half resolution.

    This is why the pixels look large and the fonts look terrible.

    It seems that if I switch to glfw it is better overall, and supports retina display.

    Fucking piece of shit GLUT


Monday Morning, September 1st, 2014:
------------------------------------

    Integrate runtime side of font code.

    Seems to be rendering a white square.

    Also seems like my ortho projection is not quite right.

    Fixed. Parameter order to glOrtho2D was incorrect.

    Now it is rendering at the expected place, but it is all white.

    Is the texture blending incorrect, or is the font texture not being generated correctly?


Sunday Morning, August 31, 2014:
--------------------------------

    Grabbed functions from stack overflow to make path, split path etc.

    Now the make fonts works even after a clean.

    Add new rule "data" that builds all data.

    Work out how to use premake to ensure that data is built before running game.

    Make sure premake actions os.exit(1) when they fail, so they don't continue
    to run the game client when the data doesn't build for example.


Saturday Morning, August 30, 2014:
----------------------------------

    Define a simple data/fonts/Font.json that describes
    the input -> output file names as well as the size

    Parse this JSON in C++ inside FontBuilder

    Add error handling if the input file does not exist, or
    if there is a JSON parse in the file.

    Iterate across the set of fonts in the JSON and print work to do.

    Actually convert the fonts over.

    Implement function to get file time as uint64_t

    Compare file times between the .ttf and the .font file and .json
    file and if the .font file is < access time of either then add
    to queue of work to do.

    Also compare with executable time so if a new version of font builder
    has been created, the fonts are rebuild automatically.


Thursday Evening, August 28, 2014:
----------------------------------

    Remove fstream and replace with fopen, fwrite.

    Sketched out FontBuilder program structure and required functionality

    Not a lot of time tonight to work on this.


Wednesday Morning, August 27, 2014:
-----------------------------------

    Implement font tool that processes the fonts in the data/font directory and 
    outputs to bin/data/fonts/... processed sets of textures for each font character.

    Implement a loader for this font, loader should pass in name

    Hack up and get the makefont basically functional, eg. outputting font.bin

    Adjust font so that it defaults to 64pixels height. Nice big font. Woo. We can afford it

    Rename to "FontBuilder"

    Create a new rule called "fonts" that invokes "FontBuilder"
    on the JSON file, running under the root directory, eg.
    bin/FontBuilder data/fonts/FontBuilder.json

    Font builder should not work one font at a time, but instead
    should work on a set of files.

    It should get this set of files from a JSON file that describes
    the font operations to perform, eg. source, dest and size of the
    font to generate.

    Find a JSON parser to use, janson is a C library, seems good:

    http://www.digip.org/jansson/

    Installed Jansson, and including it just fine.

    Did some more work to clean up C++ BS in font tool, so it is 
    fast to compile and run.


Tuesday Morning, August 26, 2014:
---------------------------------

    I very much like the idea of creating a tool that spits out textures
    from a truetype font, so I don't have to link to freetype. This is the way (tm)

    http://content.gpwiki.org/index.php/OpenGL:Tutorials:Font_System

    Downloaded "fonts for code" from here: http://input.fontbureau.com

    Created a new "data/fonts" directory for all the game fonts.

    Created a new "tools/font" for source code that converts the TTF to game font data.


Sunday Evening, August 24, 2014:
--------------------------------

    Add enough callbacks to the client so I can track stuff going on,
    eg. state changes, connect, disconnect etc.

    Make the logs nice and chatty.

    Add a timestamp to all client logs

    Hook up logs for "OnError"

    Add some callbacks on the server side.

    eg. on state change.

    Add callback for receiving client data, print out # of bytes

    Add callback for client timed out.

    Setup an instance on Amazon. Picked a compute instance so I can run the
    physics simulation on the dedi there. I'll just have to be careful not
    to leave it running as it could be moderately expensive.

    Choice of linux: Ubuntu.


Sunday Morning, August 24, 2014:
--------------------------------

    Set up separate client and server programs in game.

    -DCLIENT on client to separate client and server.

    This is the easiest way to create a client/server split.

    Cleaned up the game server create. Still not great.

    Would prefer if it was easy to subclass and setup a derived class
    that just does all the work in the constructor.

    Right now I had to create a function to create the game server instead. Sucks.

    Setup client creation too.

    Actually connect a client to the server. Verify everything is working properly.

    Add code to detect a client error and exit with error message.


Wednesday Morning, July 30th, 2014:
-----------------------------------

    Extend the soak client/server to use this same subclassed client/server
    and to exchange test context messages with test context sent from server
    to client via server data (with some additional padding...)

    This is an excellent test because there are multiple servers with
    different contexts, so connection packets are flying with radically
    different serialization in the same program.

    Works perfectly!


Tuesday Evening, July 29th, 2014:
---------------------------------

    Add interface to client and server to set context at index.

    Add a message type that relies on context to function: TestContextMessage.

    Extend stream so it's a bit nicer to get a context by index.

    Create a new unit test that sets context on the server,
    and sets this same context data as server data, and on
    the client, as soon as server data is received, sets this
    data as the user context.

    This may require subclassing the client into TestClient
    and implementing a notification callback, OnServerDataReceived

    Now switch MESSAGE_TEST to MESSAGE_TEST_CONTEXT and randomize
    the value sent within the min/max bounds of the test context.

    Need a way to get the const void ** context pointer inside the
    reliable message channel as a member, so we can use it when we
    create the test message channel.

    How can I do this?!

    Should channels have a "SetContext" method on them that they must implement?

    Is the client/server responsible for setting the context on all channels?

    Probably should be.

    Made it so that the context is a parameter input to the connection config.

    Then when the connection creates all the channels, it calls set context on them.

    Now the basic unit test passes.
 

Monday Evening, July 28, 2014:
------------------------------

    It seems a bit of a slam dunk than channel structure should be context[0]
    and client server context should be context[1].

    This has a lot of benefits, including simplification of packet factory.

    There should be context enums, like CONTEXT_CHANNEL_STRUCTURE
    and CONTEXT_CLIENT_SERVER

    There should also be CONTEXT_USER so the user knows where they
    can start adding user contexts.

    Standardized on CLIENT_SERVER_PACKET_CONNECTION as the standard name.

    Make unit tests work with channel structure as context.

    Make soak test work with channel structure as context.

    Fix protocol soak so it uses network simulator again


Sunday Evening, July 27, 2014:
------------------------------

    So it seems that the context works well.

    I think that serializing 64bit client and server guids is way overkill.

    Reduce them back down to 16 bits each. Just do rand() % 65535 + 1 (0 is special)

    Now while it's some overhead per-connection packet, it's not ridiculous.

    Note that "guid" implies globally unique id.

    There is no guarantee that the guid is globally unique.

    Should probably be renamed to "clientId", "serverId" IMO.

    Done.


Sunday Afternoon, July 27, 2014:
--------------------------------

    Seems that both client and server should have a client server context object.

    That this object should be int numClients, address, per-client, client and server guid.

    This struct should provide quick find client by address, guid lookup.

    It is basically a hot-cold split, and something we would want anyway.

    Don't do a callback, just let the connection packet inspect this client/server
    context and do its work. I think there would need to be exclusion potentially
    while accessing this client/server context but it should be changed very
    infrequently on the client/server side (connecting clients) and on the serialize
    side it would only ever be read only.

    Put a magic number at the top of the client/server context. 

    This way you can verify it is really a client/server context.

    Add basic functions like find client, add client, remove client.

    Add context to client class.

    Add context to server class.

    Switch all find client index functions over to use context on server,
    so I know it's really working. Unit tests still pass.

    Put the context array void*[MaxContexts] inside the client and server.

    Clear all contexts to nullptr initially, and set the const void** on 
    the network interface as soon as the client/server is created.

    Set the client/server context as entry 0 in the array, for now.

    Move the client/server guids back into the connection packet.

    After serializing client and server guids, if no matching client is 
    found, discard the connection packet by setting client and server guids
    on the packet to 0, and returning from the serialize.

    To implement this will need a search function that takes only
    the client and server guids, and does not care about the address.
    
    Make sure all unit tests and soaks still pass.


Sunday Morning, July 27, 2014:
------------------------------

    Now work out how we can apply this to client/server
    to only allow connection packets through with the 
    expected client and server guids.

    I think there should be a client server context.

    This should be a basic struct, it should contain the client and server guids
    and a bool indicating whether or not a connection has been established.

    Add client/server context: ClientServerContext.h

    Add a method to network interface to set context for serialization.

    Made get/set context const safe.

    Inside BSDSocket take the context saved and apply to read and write
    stream when serializing a packet.

    IDEA: It might be nicer to set an array of contexts on the stream
    vs. setting each context individually on the stream.

    This means the stream itself doesn't really "own" the array
    of contexts, it could be owned by the client/server and updated
    behind the back of the network interface.

    I think this is a much more flexible setup.

    This would allow users to modify via client/server interface
    the set of contexts on the fly, which is needed, vs. having to
    call out to the network interface, which calls out to the stream
    and so on...

    Adjust the context system to work this way.

    Update the test_stream_context function to do this.

    Update NetworkInterface.h and BSDSocket.h/cpp to work this way.    


Saturday Evening, July 26, 2014:
--------------------------------

    Exchange messages between client and server to make sure protocol
    is working properly between clients and server.

    Send a message from client to server, and on server reply back.

    On client verify that all replies come back in order, and with
    expected sequence so we know message contents are correct.

    If no message received on client for a second then error.

    Add some packet loss so shit gets real (say 50%?)

    It's annoying because adding simulation on top of the network
    interface is difficult.

    I feel like simulation should almost be done at the network
    interface level, or at least, there should be some easy way
    to hook in to it at the client server level.

    Perhaps the client and server should own a simulator object?

    Man this is annoying.

    Perhaps you could just set a simulator on a network interface,
    and the network interface would take care of the rest?

    This seems nice, but complicates the network interface impl.

    It's already pretty complicated.

    Easiest solution. Put the network simulator down as the network
    interface to use for the client and server, and pump it then
    send the actual packets on to the real network interface.

    This is possible because network simulator implements network
    interface.

    If no clients are connected for longer than... 10 seconds, then this 
    should be an error. This helps detect if the client/server connection
    is not being established properly (eg. right now, it is not, because
    I redirected to network simulator)

    Now hook the simulators up so they actually send packets!

    Tried. But it's not working well. The way that the simulator
    works only really applies to sending packets, but if the network
    simulator is passed in as 

    Seems that putting the simulator into the client and server
    objects might be the best way to do it.

    This seems lame, but I really want to keep total flexibility
    with the simulator. I'd not want the simulator to always buffer
    and then send packets over UDP. Often I want the simulator
    to work by itself, and then just directly pass packets through
    without going anywhere near sockets.

    Yep. Added to client and server directly, and now the soak
    client/server is stable under packet loss and latency. 


Saturday Morning, July 26, 2014:
--------------------------------

    Need a way to identify a client slot on the server.

    If client had an accessor for client guid, server guid,
    and the server had a public function to find a client by
    client guid, server guid and address.

    Then I could look up a client slot on the server (or -1 if not connected)

    On connect verify that the client and server data are correct.

    Pass over code and clean up


Friday Morning, July 25, 2014:
------------------------------

    Back to client/server soak test.
    
    How to detect when things are broken? 

    Detect connect and disconnect via state and print it.

    The soak test seems to reach a steady state where clients are continually
    trying to connect, but I see no clients disconnecting from servers.

    This indicates to me that the servers are getting stuck in a state
    where they are no longer able to accept connections.

    It seems that this happens even with a sleep, so it's not socket starvation.

    What is happening? Are client slots not timing out or something?

    It definitely seems like a bug!

    Note: Still happens even if I sleep for 10ms on each iteration!

    Server client slots are stuck in "Sending challenge".

    What state are the clients in?

    Mostly disconnected.

    Theory: clients are timing out immediately after connect because the
    last packet receive time is so old (from a previous session).

    Make sure to bump the last packet receive time to the current time
    at the point of connect, so it doesn't time out right away.

    Yes indeed. That was it! :D


Thursday Evening, July 24, 2014:
--------------------------------

    Passed over unit tests and removed sleeps. Fixed a bunch of stuff.

    Next, it seems that eventually a sleep is required, but not for common case.

    Solution should be, if iterations exceed 256 for common loops, sleep 1 ms.

    This should be fast in the common case, but won't timeout and break tests
    when UDP packets get dropped / queued up by the OS.

    Goal is to get the test looping infinitely at high speed, eg. run overnight.

    It would seem that I should have a sleep function in common, and have
    platform specific sleep implementation. Fuck std::chrono it is overengineered!

    Added a neat function to sleep 1ms after 256 iterations. Sorted!


Wednesday Evening, July 23, 2014:
---------------------------------

    Sketch out new soak_protocol and soak_client_server in premake.

    Implement a client server soak test.

    In this test there should be n clients and m servers.

    Sketched out a bunch of servers.

    Make sure I don't leak memory.

    Sketch out a bunch of clients.

    Clients randomly try to connect to a server by index.

    Clients randomly disconnect from a server while connected.

    There should be more clients than there are available slots.


Monday Evening, July 21, 2014:
------------------------------

    Now get client data working.

    It's passing.

    Make sure to handle client data too large on the server, eg. check for error state on the recie24ver.

    Next, client data and server data in the same test.

    Cleaned up the code a bit, sped up compile time.

    Added test for reconnect with server and client data blocks.

    Add a multiple client test with server and client data blocks,
    eg. connect max numbers of clients, each client with different
    client data and verify the server has the correct client data
    for each client.


Monday Morning, July 21, 2014:
------------------------------

    Sketched out the like cycle of data block sender and receiver in Server.h/cpp

    Now need to make sure that they are cleared properly in disconnect.

    Update the sender inside SERVER_CLIENT_STATE_SENDING_SERVER_DATA

    Hook up get client data on server to redirect to receiver.

    Hook up to process ack packet on server.

    On server hook up client server info at the earliest point it is known,
    for example, on the server side this is probably on the transition to
    the sending challenge response state.

    Now on the client create the same sender and receiver objects.

    Set the client server info at the point where the client and server
    guids are known (eg. when connection is pending and we start sending
    the challenge response back)

    Hook up the get server data to the receiver.

    Noted that I probably want the client/server data ASAP, and perhaps
    before the connection is strictly established (it may be required
    to actually serialize )

    Hook up the code to process the block fragment to go to receiver.

    Actually works now.

    Added code to handle data block reciever and set error on the client.

    Both server data tests pass now.


Sunday Evening, July 20, 2014:
------------------------------

    I'm not happy with the complexity of block fragment send and receive
    code that is cut and pasted between client and server.

    To resolve this, create new helper classes "BlockSender" and "BlockReceiver".

    The client will still have to do the actual packet payload, but
    these classes will handle all the tricky allocation, all the logic
    and can be reused across client and server (and for reservation data)
    trivially.

    Data block sender class is finished.

    Wait, there is more. Extended to include ProcessAck function.

    Now need a data block receiver class.

    Sketching it out, a lot of params required for ProcessFragment function.

    In order for the receiver to know when the send is completed, it is
    necessary to include the number of fragments in the fragment packet.

    In order for the receiver to be able to pre-allocate the receive fragments
    it is necessary for the sender and receiver to agree on the fragment size,
    this way the reciever can determine the maximum number of fragments
    from the max data block size.

    Implemented data block receiver.

    Over to unit testing now.

    Found an error in the calculation of the last fragment size.

    Create test classes that derive from the data block sender and receiver.

    Setup a test without packet loss.

    Verify block size and contents are correct.

    Setup a test with packet loss (50%). Make sure packet loss applies to 
    both directions, eg. fragments as well as acks!

    Now rework the client and server to use the data block sender and receiver
    instead of manually implementing this logic. This will require deriving
    new classes for the sender and receiver per-client and server.

    Stripped out or commented out the old code.

    Now to implement the sender and receiver classes that the client and server should use.

    Where do they go? I'm guessing a ClientServer.h/cpp might be the right place?

    Decided on ClientServerDataBlock.h/cpp

    Added ClientServerInfo struct to pass a bunch of data from Client/Server
    needed by the block sender and receiver -- eg. SetInfo. This needs to be
    done once the client guid, server guid and address are known for that client.


Sunday Morning, July 20, 2014:
------------------------------

    Design the server interface for getting client data, eg. should be same as client
    interface for getting server data, but accept a client index.

    If the client is not fully connected, make sure the client data is null.

    Should the same thing be done for server data on the client? I think so.

    Disable the m_lastPacketReceived time while in the receiving server data
    state, as this will ensure that the connect times out if it takes longer
    than this time for the data block to come down. This is intended.

    It seems silly that we don't send the client and server data at the same time. 

    Should probably send at the same time, and have the client just decide that 
    it has finished sending its data, and when that is true, send "ready for connection"
    packets, and the server can do the same, send "ready for connection" once it has
    finished sending its data.

    When the server is in the ready for connection state, and receives a ready for
    connection packet from the client, I think that would be the condition that takes
    the server to fully connected.

    This means that I should rename the concept of "request client data" and make that
    server, ready for connection instead.

    If I start by doing this, I think it will be less work than to implement and
    refactor the client data sending state machine after it is working.

    So lets start here.

    Restructure the state machine so there is no "waiting for client data" state
    on the server, only a concept of "server ready for connection" state, with
    ready for connection packets sent down to client instead.

    Now make sure all the existing tests pass. 

    To do this I need code on the server that detects that the client
    is ready for connection, and the server is in ready for connection
    state, so in that case -- go to connected state.

    Also fixed a bug where server client data was not being fully cleared
    between client connections, meaning that reconnect would fail.

    On the client side:

        a) the number of fragments in the block
        b) the ack array for the fragments
        c) num acked fragments so we know when finished sending

    What data structures do we need on the server side per-client?

        a) memory to store the client data
        b) size of the client data
        c) block (so it can be returned to caller) &block

    Implement the server side first, it's easiest.

    Added client data block per-client on the server.

    Implemented accessor Block * GetClientData( int clientIndex )

    Added code to allocate and free the per-client data block on the server.

    Now implement the client side. Needs to be an array of fragments,
    calculation of num fragments, and num acked on the client side,
    and the client needs to go into "sending client data" state and
    send this through, using the same logic that the server is using
    right now.

    Client needs fragment index. Added.

    Client also needs accumulator, eg. time last fragment sent.

    Implement update function "UpdateSendClientData"

    Actually, client needs a whole lotta stuff.

        int fragmentIndex;
        int numFragments;
        int numAckedFragments;
        double lastFragmentSendTime;
        uint8_t * ackedFragment;

    Added new function UpdateSendClientData

    Need to make sure client goes into sending client data state if
    the client has data to send. Right now it does not appear to be.

    Fixed. Also removed superflous "block" in client config.

    OK. Finished converting over the logic for client sending fragments.

    Appears to be working properly, eg. client is cycling across fragments
    and looping with modulus # client data fragments.


Saturday Afternoon, July 19, 2014:
----------------------------------

    Added data block fragment, and data block fragment ack packets.

    So there is the question of how large the packets should be for sending.

    Leave this up to the sender. I suggest a default of 1k fragment blocks
    (MTU is 1200) so this would be nice and dandy with plenty of overhead.

    Added fragmentsPerSecond to both client and server send rate.

    For the moment, it is set to 60 fragments per-second.

    This means that 64k would take just over a second to transfer down.

    This is a very good argument for compression (LZ4) of the client and
    server data blocks.

    Server needs this member data:

        a) number of fragments in the server data

    Server needs data per-client:

        a) last fragment send time (initially zero)
        b) an array of uint8_t initially memset to zero
        c) an index into the current fragment id (where search for next send starts)
        d) number of acked fragments. this tells us when to stop!

    Server needs to create the fragment acked array in constructor, per-client,
    next to the place where the connection is created.

    Server also needs to clean it up, because the ClientData struct doesn't
    have enough context to actually perform the cleanup.

    Now implement the logic to send fragment

        1. Detect if enough time has passed to send a fragment

        2. If it has, then set last fragment send time and construct a packet

        3. When constructing a packet, start at fragment index and seach for the first
           unacked packet. Assert that the number of acked packets is < the number of
           fragments.

        4. Construct and send a data block fragment packet to the client.

        5. Go to the next fragment index so sends cycle across different packets.

    Added MaxFragmentSize = 1024 in Constants.h

    I want this because I want there to be a maximum amount on the fragment
    bytes being passed in for allocation, so packets can't be constructed
    maliciously to create gigantic fragment memory blocks.

    I don't think it's worth the effort right now to do a dynamic max fragment,
    so I just added a MaxFragmentSize in Constants.h instead.

    Extend the test packet factory to create the new packet types.

    Add code to actually construct a fragment packet and send it to the client.

    Now implement the logic to on the client to process the data block fragment packet.

    Client should take the packet and throw it away if it doesn't match guids.

    Client should also throw away the packet if the fragment size is greater
    than MaxFragmentSize (constant).

    If the fragment id is out of range, then the client should throw it away.

    Otherwise, client should reply with an ack packet.

    Make sure that the fragment start offset + fragment bytes in memory does not go past size of block in bytes.

    Actually copy the fragment data into the server block on the client.

    Verify that the unit test passes. It does! Yes!

    There should be an error on connect if the config max block size is not large enough.

    Add a unit test to verify this error triggers when expected.


Saturday Morning, July 19, 2014:
--------------------------------

    Add code on server to send server data block to client.

    First up you want to include a server block in the config. This block can be null.

    Should the server take ownership of the block? Probably not. Lots of other
    pointers passed in to server config don't confer ownership. It is nice this
    way because we don't need to worry about allocators.

    There should also be a config on the client side for the maximum server block size.

    Client should then pre-allocate a block of data for the client to store this data,
    and have a int m_numDataBytes which starts out as zero. This way as block fragments
    come in from the server.

    Client should present the data as a "const Block*" so all block usage is consistent.

    Implement code on server to process the ack, but only if we are in the sending server data block state.

    Each ack being processed, check if it is in range very carefully.

    If in range, then mark than fragment as acked and increase num acked fragments.

    If the num acked fragments equals number of fragments, go to next state (requesting client block)

    Client needs to know the fragment size being sent down (for validation)

    Client needs to know total block size.

    We probably need to know the number of fragments in each packet, eg. x/y for validation.

    Need to separately specify the fragment size, and the fragment bytes in the packet (may be different)


Tuesday Morning, July 15, 2014:
-------------------------------

    Send a "Disconnected" packet from client -> server on disconnect,
    so the slot gets freed up a bit quicker than if it timed out.

    Handle the disconnected packet on the server side.

    Add a test that disables timeout and verifies that client disconnects
    properly on the server, without timing out, eg. that it receives the
    disconnect packet.

    Note that in order for this to work the client must continue to pump
    update after disconnect, otherwise the disconnected packet is not sent.

    Todo: Simplify channel structure so that it doesn't require std::function

    eg. give it the same treatment that was given to the factory.

    Boom. That removed the last STL usage and now compile times are *FAST*!!! :D

    Now that I have a .h/.cpp split, the way I have structured
    the bitsquid containers means that I am not getting the full
    benefits of short compile times.

    Split back up into Types.h and clean up the rest of the bitsquid code!

    Added Common.h as a precompiled header. Even faster.

    Disabled exceptions via "NoExceptions" in premake.

    See what "Profile" looks like now in release build.

    Looks great. Is primarily bound by allocation and free operations.


Monday Evening, July 14, 2014:
------------------------------

    Replace the <queue> usage for "PacketQueue" with something a little bit more intelligent.

    Remove <map> usage inside factory.

    Remove lambda and std::function usage as well.

    Instead of using lambdas, just get normal and make factory a base -- you override
    the create and destroy functions to implement the factory.

    Remove Factory.h. It adds no value.

    Update the code to work with the new packet factory.


Sunday Evening, July 6th, 2014:
-------------------------------

    Add a "check" macro that works in release.

    Verify unit tests pass in release build!

    Clean up or remove the resolver via #ifdef. I don't like including STL

    Added PROTOCOL_USE_RESOLVER so it can be compiled out.

    Reduce headers further inside Common.h and make todo notes
    where things need to be cleaned up.

    Add a runtime test that verifies the PROTOCOL_BIG_ENDIAN is correct.

    Grab the big endian check from sophist.h and anything else useful.

    Remove the sophist Platform.h. Nothing else useful in it.

    assert should become "PROTOCOL_ASSERT". This needs to be redefined
    to call a user specified assert function.

    check should become "PROTOCOL_CHECK"

    Code is ugly but, hey that's how it goes when you make a library.

    Remove microallocator. It's not what I need

    Now move the function for assert and check handling into the cpp
    file so we don't need to rely on stdio/cassert in the include.

    Fix the bitpacker to do bswap when necessary for big endian.

    I need to provide my own efficient byteswap functions in Common.h,
    eg. fast_ntonl, or just bswap if big endian in/out.

    Called these network_to_host and host_to_network respectively.

    Marked places that need conversion to bitsquid classes.

    There are only too places aside from resolver, this shouldn't be too much work.


Saturday Morning, July 5th, 2014:
---------------------------------

    Run through the profile in release build now. How does it look?

    Looks pretty good. There is still a malloc cost, this could be
    spill-over from the scratch allocator, but is more likely the
    message, packet and block allocation going through the malloc
    allocator.

    To actually measure we'll need a custom pool allocator for messages.

    Prefix all #define names with PROTOCOL_, eg. PROTOCOL_PLATFORM_WINDOWS

    Fix all #defines to be prefixed with PROTOCOL_

    Moved all preprocessor configuration into Config.h


Thursday Morning, July 3rd, 2014:
---------------------------------

    Convert packet factory to use custom allocators instead of new.

    Convert message factory to use custom allocators instead of new.

    Is there a bug in the scratch allocator? There probably *isn't*
    but the crash was due to memory leak/corruption on my part.

    Test going back to the real scratch allocator. Does it work now?

    Yes it works now.


Wednesday Morning, July 2nd, 2014:
----------------------------------

    Pass over unit tests and convert remaining new/deletes.

    Channel structure should have a corresponding destroy channel
    function, for maximum flexibility of implementation, eg. could
    actually new the channel if it wanted to.

    Channel structure should create channels via base allocator passed in.

    Update code to call DestroyChannel on channel structure instead of delete.

    Had to split up the channel and channel data allocators, as one of them
    typically will want to be a scratch allocator and the other is long term.

    Unit tests pass with all channel and channel data allocations going
    through custom allocators!


Tuesday Evening, July 1st, 2014:
--------------------------------

    Pass over code // todo

    Removed TestNetworkInterface.cpp. Was not adding value.

    Pass over code and mark each new that needs to be converted to custom allocators.

    Fixed double free introduced in network simulator.

    Converted BSDSocket to use allocator passed in to config.

    ^-- Falls back to default allocator if null allocator passed in = NICE

    Convert sliding window to accept allocator in ctor.

    Convert sliding window tests.

    Convert classes that use sliding window: Connection, ReliableMessageChannel.

    Convert remaining fragment allocations inside ReliableMessageChannel.cpp

    Probably want an allocate array macro? eg. create an array and then
    initialize each array entry to the default ctor via placement new?

    Yep. Added PROTOCOL_NEW_ARRAY and PROTOCOL_DELETE_ARRAY.

    Convert remaining allocations in channel to use scratch allocator.

    eg. the channel data create etc.

    Make sure matching delete is replaced with PROTOCOL_DELETE

    Had to update channel structures, they were still using new.

    What other allocations can be converted?

    Network simulator needs custom allocator passed in. Relatively easy.

    What else is easy?

    Client needs to be converted.

    Server as well.

    Split out allocator base class into Allocator.h and removed
    #include "Memory.h" from headers.


Thursday Morning, June 26th, 2014:
----------------------------------

    Remove the need for AF_INET6 etc. in DNSResolver.h and BSDSocket.h

    Can just be replaced with "bool ipv6" really. I don't know that I care
    about the transition which is either. That's not well tested.


Wednesday Evening, June 25th, 2014:
-----------------------------------

    Fixed some cases where packet leaks were caused by delete instead of destroy.

    Track down why we are leaking packets.

    Leaking now seems to be *triggered* exclusively by delay.

    Packet leak is fixed, but now I have a message leak.

    Message leak only occurs 100% with packet loss and latency,
    it however does occur *occasionally* with just packet loss.

    It seems that this is now an actual message leak.

    Probably something to do with the processing of acks,
    or handling of messages between packet <-> queues.

    Refs are added in only two places:

        a) On create
        b) When adding a message to a packet

    Should probably also add a ref when taking a message from packet -> receive queue.

    Seems that the leaking messages are coming from packet -> receive queue?

    I think the packet factory doing the freeing is not the same as the one creating the packets.

    Nope. I've got it! It's when a message comes in, but the message id
    has already been added to the receive queue. In this case, it is clobbering
    the previous message in the sliding window.

    FIXED!!!!

    Now bring it all back...

    Get the small block test passing.

    Get the large block test passing.

    Get mixture of message types unit test passing.

    Looks like I have a double message delete somewhere in the mixture test?

    Nope. Was just creating the test message with new.

    Get remaining unit tests that were disabled passing again.

    Blocked now at test_bsd_socket_send_and_receive_ipv4

    My guess is that we are creating a packet via new and passing in?

    Some crazy stuff going on with copy constructor / assignment?

    Seems that serialize is not getting called at all for connect packet?

    Sorted. Just some crazy stuff with assignment.

    Now, fixed some memory leaks spotted in BSDSocket.cpp -- on destructor
    send and receive queues needed to be cleaned up, and on send packet
    if in error state, it was leaking the packet.

    Get the client/server tests passing.

    Seems that we have some old packets created via new in Client.cpp

    We had that, but still there are problems.

    Seems that we have multiple packet factories active.

    eg. a packet created with one factory, destroyed with another...

    Something in client is causing this.

    Perhaps the network interface has a different packet factory
    than the client/server?! Somehow?!

    Seems like the client needs to get the packet factory
    from the network interface, not the other way around.

    Fixed.

    Now the disconnect packet was being leaked on client. Fixed.

    All unit tests pass again!

    Reduced header dependencies. 

    Moved socket stuff into BSDSocket.cpp etc and out of Common.h

    Get the soak test passing.

    Pass over and remove unnecessary debug logs.


Wednesday Morning, June 25th, 2014:
-----------------------------------

    C++ guarantees that stack objects are destroyed in reverse
    order of their construction, so I don't need to worry about
    the message factory being destroyed before the packets and
    other structures that point to messages.

    Repeated iterations on the small block test still fails.

    But now, paydirt -- it is a message that is not being released.

    So we know now that it is caused by a message not being cleaned up...

    The question is. Why does this happen in the small block test but
    not the normal message test?

    Answer: The leak still happens for the non-small block message test.

    I'll start here then, and keep it separate from blocks.

    Is it packet loss related then? I think it must be.

    Complete packet loss isn't enough to cause it.

    It must be something ack related. Perhaps it happens when a
    sliding window entry is *reused* to insert another sequence?

    In this case, the message may not be getting cleaned up.

    I'll bet this is it!

    When a state was added to simulator, it was not immediately switching
    to that state, but random change it would switch to it -- this was why
    the packet loss wasn't triggering in the common case for short tests,
    and why repeating the unit test was necessary to reproduce the leak.

    Fixed it so that if you add a state to the sim, the first state 
    you add is immediately active, eg. 90% packet loss now active
    in all reliable message channel tests.

    Memory leak repros 100% now. It's definitely caused by packet loss.

    Why would the message be leaking? Are packets not getting cleaned up?

    Add packet leak check to packet factory.

    Convert packet factory to have specialized leak detection as well?

    eg. centralize packet create and free to go through the factory?

    Yes. Done.

    Confirmed. We are leaking packets, not messages!


Tuesday Evening, June 24th, 2014:
---------------------------------

    Convert std::string to c_str in Address class

    Remove std::string usage from Client class (hostname)

    Added MaxHostName const (256).

    Remove std::string from Channel class (channel name)

    We are getting memory leaks in small message test.

    What is leaking?! It's random, doesn't happen 100%.

    Confirmed: Dropped packets are triggering the loss.

    Is there something wrong with the message cleanup?

    I think we have an error where the delete on the *read*
    packet does not clean up the messages it contains.

    This happens on packet drop in the simulator.

    However, if the packet is actually processed by the connection,
    the connection takes those messages and inserts them into recv
    queue, where they will be received.

    It seems that a packet must be put into a state after being read
    that each of the messages it contains will be deleted, but upon
    processing this packet, those messages must be marked not to be
    deleted when the packet is deleted (which it will be).

    In short, the packet must transfer ownership of its messages
    to the receive queue when it is processed, but otherwise should
    delete those messages.

    Yes. That was it. It's not leaking anymore.

    Nope. There is another memory leak still happening.

    To reproduce this leak requires the simulator having latency.

    It still happens with zero packet loss. 

    It doesn't happen if there is no simulator latency.

    Theory: If a packet is deleted while in the simulator it is not deleting the block.

    No. It's not this simple. Just deleting the read packet before processing it
    doesn't reproduce the leak.

    Analysis shows however that the block that is not being freed,
    is a block that was *sent*.

    So how could a sent block not end up getting deleted?

    It is inserted immediately in send queue

    It should be deleted as long as:

        - remove ref is called on send queue entries
        - remove ref is called on packet destructor
        - all packets are actually deleted

    I think messages are a cluster fuck right now...

    They should be uniformly ref counted, most likely.

    It seems that special cases of the messages to not be ref counted
    are causing problems, eg. complexity, asserts where messages are
    deleted with ref count != 0.

    Message factory could have a method "Release" which decreases
    ref count for message, and deletes that message if its ref count
    becomes zero.

    Deleting messages should become "outlawed". They can only be released.

    Sketched out the new "MessageFactory". Messages now start with 1 ref.
    It is illegal to delete a message manually now, you must release.

    Designed the process for messages going *in* being sent, I think they
    have one ref, and this ref just transfers to the send queue at this
    point.

    Each time a message is included in a packet, add ref.

    Each time a packet is deleted with messages in it, release.

    When a packet is processed, grab the messages and add to receive queue
    and add ref.

    When messages are recieved, remove from receive queue and pass
    to caller.

    Caller is now responsible for calling release on each message.

    If messages are in send or receive queue on reset, release message.

    This should avoid leaks and complication of messages entirely!!!

    Messages *always* need to be released. vs. sometimes needing release
    and other times needing delete.

    Remove block message ctor that accepts block. Cannot be used.

    Make messages dtor protected, and friend the message factory class.

    Make message addref/release protected. Must always go through factory.

    You *must* use the factory for deletion of messages.

    Remove cases where messages are created via "new". Always
    create messages via the factory. This will make it easier
    to convert the factory to use a custom allocator.

    Convert reliable message channel code to use uniform addref/release
    on all messages.

    Convert over the simplest test for reliable message channel,
    eg. non-block case to use the updated message addref/release.

    Works.

    Convert over the small block test.

    Still leaks.

    Simplified the test as much as possible while still reproducing the leak.

    Seems that if a packet is serialize read containing messages,
    these messages are not being properly released on packet delete.

    Why is it leaking?

    Was as simple as an extra addref on send message that didn't need to be there.

    Now there is another leak when the "real test" is turned back on.

    Is it packet loss related?

    Does not happen if packet loss and latency are disabled.

    Does it happen with latency only?

    No. It seems to be specific to packet loss.

    How would packet loss cause a message leak?

    Trust nobody. 

    Add message factory checking, eg. a map that verifies all messages
    are freed in destructor, and that all messages being freed are known
    pointers created by the factory.

    Fixed up duplicate message factories, and the message alloc and free looks good.

    Now the small block test passes!


Monday Evening, June 23rd, 2014:
--------------------------------

    Change tack and start with the lowest level, "serialize_new_block"

    Now switch in new block for old and get the library compiling.

    Now update the unit tests to work with the new block.

    Converted small block test.

    Converted large block test.

    Converted mixture test.

    Fix Soak test to work with new blocks.

    Verify soak test is stable.


Monday Morning, June 23rd, 2014:
--------------------------------

    Implement "Block" manually, don't use std::vector.

    Block doesn't need to be resized. It just needs a uint8_t *
    and a size in bytes. It *owns* that data, so it needs to be
    able to delete that data in dtor.

    Should block own the data? 

    I think it needs to, because inside a message, it is expected
    that a message ownership is passed to the channel on send,
    and back to caller on receive.

    So are blocks copyable?

    No. Not by value.

    How can you take the data in a block and store it somewhere else?

    You need to copy it! Blocks are messages and are temporary. 

    You cannot hold onto block memory, this will fuck up the life cycle
    of the block allocator.

    If you receive a block and want to hold on to it for a while
    after you process the message, you'll need to make a copy of it.

    Ideally, blocks should be received and processed into some other
    form, eg. deserialized, decompressed and so on.

    Created "NewBlock" class which allocates and frees using an allocator.

    Renamed "Deallocate" to "Free". Deallocate is long.

    Need three allocators:

        1. Message allocator
        2. Small block allocator
        3. Large block allocator

    These are "fundamental concepts" in the library,
    so I don't see the problem with specializing like this.

    Could just have the channel structure be the place that owns these.

    Added "SetAllocators", "GetMessageAllocator", "GetSmallBlockAllocator"
    and "GetLargeBlockAllocator" to channel structure.

    By default all allocators point to default allocator (malloc)

    Messages should not need a copy of their own allocator. They 
    must be passed to the correct allocator on destructor.

    Blocks are different. They need to implement their own dtor,
    and thus they must know what their allocator was to clean
    themselves up.

    Only other option would be for block allocator to be a runtime
    type thing, but I don't like that.

    A guiding principle is that you should be able to have two
    differently configured connections running in the same process
    with different channel structures, different message sets and
    different allocators -- without any interference or problems
    between these two systems.

    Added small test suite for block. Create it, memset zero, free it.

    The small block serialize will be a problem. If it is a small block
    the serialize is going to need to be specialized so that it knows
    the allocator to use.

    This means that small blocks cannot just serialize automatically
    anymore, they must have a special serialize fn and the channel data
    serialize must *know* how to serialize small blocks.

    Note that blocks themselves also need an allocator, eg. we create
    blocks by pointer and pass them around this way. In effect, blocks
    are like messages.

    It might be a better idea to specialize the block message such that
    *it* owns the block entirely, perhaps by mixing in the block interface.

    I don't like the idea of suppyling a separate block allocator for
    the block class itself!

    Sketched out "NewBlock" class.

    Added default ctor for uninitialized block.

    Updated dtor to be able to handle not having valid data. Do nothing.

    Added method "Connect" which takes allocator, block and size,
    and "Disconnect" method. This allows transferring ownership
    of a block of memory from one block instance to another, or 
    even externally from the block.

    One tricky part about this is that blocks that are passed in
    to "SendBlock" in reliable message channel will actually have
    disconnect called on them inside that method, meaning the block
    is invalid (IsValid = false) on return.

    I can live with this.

    Need to work out how to perform small block serialization.

    My guess is that I just need a special serialize method which
    takes the allocator as a parameter. This can be done if I check
    each message is a block, cast, and call a specific block serialize
    method in that case.

    Assert false in the dummy serialize then.

    *ALTERNATIVELY*. Could stash the allocator inside the block instance
    inside the block message, and then on serialize read, rely on this
    to get the block message.

    Seems a bit dodgy though. It's still a "special" case that needs
    to be performed when serializing the small block. One special case
    vs. another. Take the simplest one?


Sunday Evening, June 22nd, 2014:
--------------------------------

    Pass over remaining std::vector and see if they can be removed easily.

    Cleaned up vector usage in resolver.

    Greatly simplified resolver by removing callback support.

    Refactored so DNS resolver doesn't allocate result, returns by value.

    Updated tests to work with new resolver interface.


Friday Evening, June 21st, 2014:
--------------------------------

    Get SoakTest.cpp building, eg. pm soak (done)

    Get UnitTest.cpp building, eg. pm test

    Move test message definitions into "TestMessages.h"

    There are problems where multiple definitions of "PacketFactory" 
    are causing linker errors and weirdness in tests, make sure that
    only one "TestPacketFactory" is declared in "TestPackets.h"

    Maybe rename Sockets.h -> Network.h (it's not really SOCKETS per-se...)

    There is a crash on malloc/free inside SoakTest.cpp now

    What happened?!

    Was incorrectly using the old client/server packet factory,
    when the code was expecting it to be using the test packet
    factory instead.

    There is a rare crash in client/server tests

    Added a loop mode for UnitTest.cpp to catch it in the debugger.

    Looks like it might be a memory trash. 

    The message this pointer is garbage: 0x071973e753a50afa

    Run valgrind over the unit test, and yes, it finds some bad stuff:

        valgrind: m_mallocfree.c:303 (SizeT get_bszB_as_is(Block *)): Assertion 'bszB_lo == bszB_hi' failed.
        valgrind: Heap block lo/hi size mismatch: lo = 1088, hi = 4608533498775470080.
        This is probably caused by your program erroneously writing past the
        end of a heap block and corrupting heap metadata.  If you fix any
        invalid writes reported by Memcheck, this assertion failure will
        probably go away.  Please try that before reporting this as a bug.

    My guess is that the error is *specific* to that test.

    Or, it is something that only shows up with multiple clients.

    Yes, there was an invalid index access because of a fall through
    when denying a connection do to being full, it was not returning
    and then would access m_clients with index -1.


Thursday Evening, June 20th, 2014:
----------------------------------

    Convert to a premake based build system.

    Convert to .h / .cpp split

    Address.h
    BitPacker.h
    Common.h
    BSDSocket.h
    Connection.h
    Client.h
    Server.h
    DNSResolver.h
    NetworkSimulator.h
    ResolveWrapper.h
    Channel.h


Sunday Evening, June 15th, 2014:
--------------------------------

    "BSDSockets" should become "BSDSocket"

    Move protocol id handling entirely to the interface layer

    Remove protocol id concept from Client.h, Server.h and ClientServerPackets.h

    If a packet comes in over the interface with wrong protocol id, discard it
    immediately without even trying to create a packet of type.

    This allows different interfaces to handle protocol id, whether they
    need it or not (some will not!).

    Plus reduces cost of processing packets dramatically, if a stray
    packet comes in it is read, and discarded immediately before any
    serialization is done.


Sunday Afternoon, June 15th, 2014:
----------------------------------

    There is an assert in SoakTest.cpp to do with block receive.

    Seems that block size on receive might be off. I probably broke it?

    Fixed. Was not sending a large block when expected.

    Try to get valgrind running again, eg. with "-g"

    Can it find any leaks now?

    No. Valgrind thinks nothing has "definitely" or "potentially" leaked.

    Standardize counters to BSD_SOCKETS_COUNTER_... etc.

    Move all enums to Enums.h

    Move all packet definitions into "Packets.h"


Sunday Morning, June 15th, 2014:
--------------------------------

    Removed std::vector usage from channel structure.

    Add Constants.h and put stuff there that are static constants defining library max

    Set MaxChannels = 8.

    Replace with static array of size MaxChannels

    Remove the dynamic allocation of per-channel data in ConnectionPacket etc.

    Remove the dynamic per-channel allocation in Connection class.

    Clean up the "Reset" inside 

    Added "GetError" int to Channel so a channel can indicate they are in an error condition,
    back to the connection on update.

    Make sure channel clears error on reset.

    If any channel is in error, tear down the connection and go into an error state.

    If client notices connection is in error, disconnect and set error.

    If server notices connection is in error, disconnect and set error.

    Standardize enums to all caps, eg. CLIENT_ERROR_NONE, not CLIENT_ERROR_None

    Fixed various breakage in Address.h and BSDSockets.h that I added.

    Run across all unit tests and fix.

    TestClientServer.cpp is broken because it's not allocating foundation memory for tests.

    Fixed. All tests pass now.


Saturday Afternoon, June 14th, 2014:
------------------------------------

    My guess is that we may not be deleting *sent* messages

    How can I determine if this is the case? Log the messages
    allocated for send, and the deletes of messages via the
    destructor in the packet.

    Do they match up? :)

    Not at all. It appears that sent messages are not being deleted!

    It seems I'm just not setting the "releaseMessages" flag.

    It appears the memory leak has gone away.

    Will the random hang go away as well?

    No. Looks like the random hang is actually in the scratch allocator.

    Replace scratch with malloc allocator, and see if the bug goes away.

    Yes. That fixed it. My guess is that there is either a bug
    in the scratch allocator, or a memory trash that is benign
    for malloc, but for scratch is fatal.

    Profile results with #define PROFILE 1 (no printf)

        22.3% - Connection::ReadPacket
        19.4% - ConnectionPacket::SerializeRead
        12.6% - Connection::WritePacket
         8.0% - ConnectionPacket::SerializeWrite
         7.7% - ReliableMessageChannel::SendMessage
         3.0% - ConnectionPacket::~ConnectionPacket
         2.6% - ReliableMessageChannel::ReceiveMessage
         2.3% - ReliableMessageChannel::CanSendMessage
         0.5% - std::vector<ChannelData*> <--- REMOVE THIS BS. (DONE)

    Basically it's looking pretty good. Perhaps the ReadPacket
    which calls into "ProcessAcks" is a bit slower than it could
    be, but then again, it's doing *actual work*.

    And if "CanSendMessage" which is basically a mod operation
    and a single memory lookup is 2.3% -- then something that
    is 10X as expensive as a function that does *no work at all*
    is probably not that bad.

    It's curious why the ReadPacket and Write packet are slower
    but it's clear that the majority of the cost here is probably
    cache misses.

    See if a hot cold split for the sliding window makes perf issues go away.

    I'm thinking a 32bit sequence + flags header, separate from the actual
    data which is in a different struct. Seems like a good idea!!!

    Looking at the structs inside the sliding windows, it does not seem
    that a hot cold split would be a good idea, unless I can somehow
    reserve one of the 16bit values (eg. zero) as "not valid", but then
    I'd have to add logic everywhere to special case the sequence.

    Removed std::vector usage from Connection.h

    I need to regroup and get everything in a solid state again.

    Stop optimizing for now, it's "good enough" for both cases.

    Take stock of remaining allocation that needs to be converted over.

    Cleaned up error handling for BSD socket interface creation.

    TestBSDSockets.cpp is broken again now that I'm properly deleting packets.

    Fixed by adding template packets on the stack that can be compared
    against, so we don't use the deleted packets to check if the received
    packet value is the same as sent.


Saturday Morning, June 14th, 2014:
----------------------------------

    Now lets profile and see if it has made any improvement...

    It seems a lot of the cost is still there.

    My guess is that the std::vector of messages inside the
    sliding window POD struct is actually the cause.

    Replace this with a standard pointer.

    Done. Lets see how the performance has changed...

        91.0% - Connection::ReadPacket -> ~ReliableMessageChannelData -> ScratchAllocator::Free (???!)
         2.9% - ConnectionPacket::SerializeRead
         2.2% - ConnectionPacket::SerializeWrite
         0.8% - std::vector block allocate
         0.8% - BlockMessage::~BlockMessage
         0.2% - Connection::WritePacket

    It seems that the memory leak has been fixed.

    But the result above doesn't make a lot of sense -- the scratch allocator
    just *cannot* be that slow...

    I'm suspecting that we have an actual problem with the profiler here.

    Nope. It's a problem with the soak test. It's getting *STUCK*
    at a certain point, and when it gets stuck here packets stop
    being written/read as far as I can tell... hence the result.

    Need to debug this!

    Definitely having some problem with scratch allocator being slow.

    It's O(n) and seems to under normal usage have a lot of iterations
    updating the scratch pointer, eg. > 32 iterations.

    Is this legitimately a problem? It's probably better to not
    perform *any* scratch allocation work for the message ids in 
    the channel data, eg. we could have a pool allocator for the
    fragments, and a flat array indexed by sliding window index
    for the message ids.

    Would this improve performance? Probably.

    Scratch should only be used for things which are typically
    freed in the same frame, in my opinion, eg. temp data.

    Tried the microallocator from Jon Ratcliff

    https://code.google.com/p/microallocator/

    It appears to be slower than normal alloc?!

    Seems that the most important thing for me to do is to NOT
    perform any runtime allocations or work at all, for normal
    operation of the reliable message channel, if I can avoid it.

    This wil be the absolute fastest, at the cost of memory, which
    I have a lot of anyway!

    Remove the std::vector used for sent packet data!!! It's really slow.

    Here is the actual cost, before the bug hits, with -O3, scratch memory is a red herring...

        (printf taking up rest...)
        27.4% - Connection::ReadPacket (most of which is allocations/frees etc...)
        10.8% - ConnectionPacket::SerializeRead
         9.9% - Connection::WritePacket
         1.9% - ConnectionPacket::SerializeWrite
         1.8% - BlockMessage::~BlockMessage
         1.1% - ReliableMessageChannel::SendMessage
         1.0% - ReliableMessageChannel::SendBlock
         0.6% - ConnectionPacket::~ConnectionPacket
         0.5% - ReliableMessageChannel::CanSendMessage
         0.5% - ReliableMessageChannel::ReceiveMessage
         0.1% - std::vector<ChannelData> alloc

    Now normalizing to get back to 100%:

        Total is 55.6%, so to normalize, multiply by 100.0/55.6:

        49.3% - Connection::ReadPacket (seems a lot!!!!)
        19.4% - ConnectionPacket::SerializeRead
        17.8% - Connection::WritePacket
         3.4% - ConnectionPacket::SerializeWrite
         etc.

    So basically the important thing is to try to work out why
    Connection::ReadPacket is so slow. I don't think it's doing
    enough work to justify being several times the cost of the
    other things.

    Tried disabling mutex for micro allocation. It's now significantly
    faster than the normal malloc -- eg. 777ms compared to 2500ms malloc.

    Added MicroAlloc licence to LICENCE

    Convert remaining parts of reliable message channel to not use 
    allocations or std::vector, eg. sent packets.

    Converted the messageIds std::vector inside SentPacketData
    to be a pointer to worst case allocated flat array of uint16_t

    Now profile again:

        12.9% - Connection::ReadPacket
        11.3% - ConnectionPacket::~ConnectionPacket (???!)    --- scratch deallocate
         5.3% - ConnectionPacket::SerializeRead
         4.4% - Connection::WritePacket
         2.1% - ConnectionPacket::SerializeWrite
         1.8% - BlockMessage::~BlockMessage
         1.5% - ReliableMessageChannel::SendMessage
         etc...

    ~ConnectionPacket now seems excessively high?

    My guess is that performance *may* be depending on the particular mix
    of fragments vs. message serialization I'm getting. Might be a good idea
    to drop down exclusively to message serialization and profile that.

    At least can see the baseline without fragment silly memory walk cost.

    Yep. With just messages it's looking totally reasonable.

    Suspicion confirmed. It's the fragment walk that is expensive.

    Here is the profile with just messages being sent:

        6.1% - Connection::ReadPacket
        4.1% - ConnectionPacket::SerializeRead
        2.8% - Connection::WritePacket
        1.8% - ConnectionPacket::SerializeWrite
        1.3% - ConnectionPacket::SendMessage
        0.5% - ReliableMessageChannel::ReceiveMessage
        0.4% - ConnectionPacket::~ConnectionPacket
        0.3% - ReliableMessageChannel::CanSendMessage

    Note that we still seem to have a memory leak. What could it be?

    Also, the random stall still happens when just sending messages.


Friday Evening, June 13th, 2014:
--------------------------------

    Remove "use namespace stl" so I can have the new foundation
    stuff co-existing with the STL during the transition.

    Having some problems with the factory now. Not sure why.

    TestBSDSockets.cpp fails with factory issue.

    TestClientServer.cpp fails with some problem in the deny connect test.

    It seems the packets are not getting through, eg. they are truncated.

    The cause is the change I made to "GetBytesWritten" vs. GetTotalBytes.

    Yes. The bytes written was off and was truncating packets.

    Added a check at the end of packets to pick up truncates in future.

    Search through for "std::" and find an easy place to start converting.

    Well there is no reason to use std::vector for counters. They are static size!

    Switched BSDSockets m_recieveBuffer to a normal pointer array allocated with new
    (again, it does not need to be resized!)

    Found a memory leak. Was not deleting packets on send inside BSDSockets.h

    Converted fragments to use the scratch allocator. It's picking up leaks
    on the large block test (when they are actually being used).

    Rework fragment allocation to go through scratch.

    Found another memory leak where the simulator wasn't going
    out of scope, and was holding on to some packets which tripped
    the memory leak check.


Thursday Evening, June 12th, 2014:
----------------------------------

    Integrate the allocators from bitsquid engine.

    Brought across basic allocator and helper functions.

    Brought across temp allocator.

    Bring across malloc backed allocator and the rest of it.

    Bring across allocator tests.

    Verify stuff is basically working.

    Bring across array.

    Added a #define NOSTL 1 that is handled in Common.h to
    help with the transition away from STL

    Bring across hash

    Bring across murmur hash fn

    Bring across queue

    Bring across pointer arithmetic tests.

    Bring across string stream.

    Verify all tests pass.


Thursday Morning, June 12th, 2014:
----------------------------------

    Fixed align in measure stream to always return 7 bits.

    We can't know the alignment in the measure stream
    exactly so be conservative and assume 7 bits.

    It's possible that if the message is *always* aligned
    at start, that the align would be correct as measured
    should consider whether this is more correct.

    However, if message align is disabled, but an align
    is added inside a message then it would break without
    this change, so I think this is for the best.

    BSDSockets was actually always writing max packet size packets,
    when it should have been writing packets of the size of the actual
    data serialized.

    Fixed by adding new method "GetBytesWritten" and renamed other
    method "GetBytes" to "GetTotalBytes" to avoid future confusion.

    There is a serialize measure overflow in the reliable message unit test.

    There was an error in measurement in the overflow check of the
    measure stream. eg. multiply bits by 8, instead of test bits
    written > total bytes * 8.

    Valgrind is not giving memory leak source locations.

    Seems broken on MacOSX

    It did find a bug in relative int serialization of message ids though.

    Why is serialize read slower than serialize write?

    eg:

        28.2% - ConnectionPacket::SerializeRead
        17.0% - ConnectionPacket::SerializeWrite

    I tried removing the htonl and ntohl calls inside the bitpacker,
    and this probably made it faster, but didn't affect the ratio
    of read to write speed.

    Honestly to fix this I'm going to need a better profiling tool.

    I'll probably even need to look at the generated assembly.

    Also, make sure the compile is doing a good job of splitting
    the read/write out and generating only the code required.

    Valgrind seems to assert that it is in fact doing this.

    Now having a read overflow in soak test.

    I think there is a serialize desync and it's only picking it up there.

    Maybe it's the first 65535 wrap breaking with message id?

    Yes, it looks like inverted logic on the 65536 addition for write.


Wednesday Evening, June 11th, 2014:
-----------------------------------

    Clearly usage of smart_ptr is silly.

    I'm going to have to drop back to not using this.

    Also, should not use STL.

    Similarly, I should not use exceptions *anywhere* for error reporting.

    (Otherwise with the standard pointers I have now, it would get painful)

    The bitsquid foundation library looks like a good start.

    However, it does not include pool allocators, which is a bummer.

    I think I should take the bitsquid code as a starting point and
    then merge it in with my own code. It is a nice system, but only
    a start.

    Look through the code and prep STL usage for removal.

    Remove usage of cout and replace uniformly with printf.

    Remove exception usage and make notes where we need 
    error handling, eg. set error flag, or fatal error.

    Removed "format_string" function.

    Remove shared_ptr usage. Replace with standard new and delete.

    This should let me profile the difference, plus think through
    who owns each pointer.

    Make sure that all ReceiveMessage calls delete the message afterwards.

    Make sure all ReceivePacket calls delete the packet afterwards.

    Connection data now deletes channel data on destructor.
    
    Make sure the reliable message channel data cleans up in destructor.

    Important, it doesn't need to clean up the messages, these are owned
    by the send queue, and are deleted once messages are acked.

    Message needs to be deleted on ack (bitpacked message *and* large block)

    SoakTest definitely has some problems.

    We are deleting some memory we shouldn't:

        SoakTest(29098,0x7fff7d306310) malloc: *** error for object 0x7ff46901e200: pointer being freed was not allocated

    And there are serialization errors that make no sense, eg. corrupt data.

    Message channel definitely has problems. The same crash occurs in the unit test.

    It's breaking in the large block test.

    It's clearly something to do with fragment sending...

    But the fragment is passed in to the channel data... it is supposed to be deleted in the destructor?

    Looks like a double delete. My guess is that the channel data was getting accidentally copied somewhere.

    Yep!

    Get the soak test working.

    Profile the soak test again.

    Getting some crashes and super weird behavior still. 

    There is still somethig broken!

    Suspect the message may be getting deleted early for some reason,
    perhaps while the channel data is still using that message.

    But is this even possible? I'm not sure it actually is.

    Ohshit. The messages *are* included in the packet,
    and the packet is serialized LATER, eg. inside the
    BSDInterface.

    This is probably the cause of it. This is CRAZY. I'm not sure how to fix this!

    (one hour later...)

    I think I can fix it by adding a reference count to the connection packet.

    Basically, the connection packet has ref count 0 initially.

    Then, I can addref to message for each packet that includes it,
    as well as addref when it is added to the send packets.

    Then when it is acked, *or* when a packet is deleted, 
    decrement the ref, if the ref is zero, delete the message.

    That seems to have fixed it. Soak test works now!

    Still getting a crash in profile mode SoakTest.

    Was caused by feeding the written packet directly back in for read.

    Can't do that. Realized the soak test wasn't even running any
    serialization as well -- crazy. Fixed it.

    Now the soak test is awesome.

    Profiling the SoakTest again:

        67.4% - Connection::WritePacket (of which almost all is ReliableMessageChannel::GetData)  <--- WHAT IS GOING ON?!
        27.2% - ConnectionPacket::SerializeWrite
         0.2% - ConnectionPacket::SerializeRead (!?!?!?)

    Two things are strange:

        1. What could possibly be inside Connection::WritePacket to be so slow?

        2. Why is serialize write 100X slower than read?!

    I can't explain this.

    Bitpacked messages only:

        65.2% - Connection::WritePacket
        29.9% - ConnectionPacket::SerializeWrite
         0.4% - Connection::ReadPacket
         0.2% - ConnectionPacket::SerializeRead

    Small blocks only:

        74.5% - Connection::WritePacket
        19.5% - ConnectionPacket::SerializeWrite (makes sense, less work to do... kinda)
         0.5% - Connection::ReadPacket
         0.2% - ConnectionPacket::SerializeRead

    Large blocks only:

        46.0% - ConnectionPacket::SerializeWrite
        34.1% - Connection::WritePacket
         1.3% - ConnectionPacket::~ConnectionPacket
         0.1% - ConnectionPacket::ReadPacket
         0.1% - ConnectionPacket::SerializeRead

    *** I DON'T KNOW WHAT IS GOING ON ***

    Strangely if I turn off the "GetData" to always return nullptr:

        36.0% - Connection::WritePacket
        29.9% - Connection::ReadPacket
         7.8% - ConnectionPacket::~ConnectionPacket
         6.4% - ConnectionPacket::SerializeRead
         6.1% - ConnectionPacket::ConnectionPacket (?!)
         6.0% - ConnectionPacket::SerializeWrite
         1.0% - Connection::Update

    It seems that read and write can get very close to the same cost.

    Why is it then that serialize read and write differ so much
    under real world data?! --- OH... the optimizer might be
    working out that you aren't actually reading into data
    that is being used... DOH

    Nope. Tried this.

    I think there is something broken with -DNDEBUG

    I think the code is behaving incorrectly.

    Dropping to without -DNDEBUG and seeing what profiler looks like:

    YES! It's more reasonable.

    Also, we *definitely* have a memory leak!!! =)

        21.2% - ConnectionPacket::SerializeRead
        12.6% - ConnectionPacket::SerializeWrite
        12.3% - Connection::WritePacket
         8.9% - Connection::ReadPacket
         1.5% - Connection::SendMessage
         1.3% - ConnectionPacket::~ConnectionPacket
         1.1% - BlockMessage::~BlockMessage

    ^--- Above looks much more reasonable.

    Need to work out why SoakTest.cpp is broken with -DNDEBUG!

    Found it! A ++m_wordIndex inside an assert. Dumbass.

    This explains everything.

    Now we have:

        27.9% - ConnectionPacket::SerializeRead
        17.0% - ConnectionPacket::SerializeWrite
        12.3% - Connection::ReadPacket
        10.4% - ConnectionPacket::~ConnectionPacket (mostly vector free)
         8.9% - Connection::WritePacket (mostly vector new)
         5.9% - vector new
         5.8% - BlockMessage::~BlockMessage (mostly vector free)

    Bottom line looking at these stats -- I need custom allocators!

    Also, I think if I introduced a faster measure stream, it would reduce
    the cost of the serialize write to be at the same level as the read.

    I should do this ASAP.

    Implement specialized MeasureStream

    This should speed up the event measure quite a bit.

    Now it is:

        27.7% - ConnectionPacket::SerializeRead
        17.8% - ConnectionPacket::SerializeWrite
        12.3% - Connection::ReadPacket
        10.5% - ConnectionPacket::~ConnectionPacket
         8.7% - Connection::WritePacket
         6.1% - vector new
         5.7% - vector free


Tuesday Evening, June 10th, 2014:
---------------------------------

    eg. add a mode where the BSDInterface isn't created and packets
    and just read and written without going over sockets.

    This will allow for greater focus on the actual packet processing
    vs. being CPU bound and sleep bound for sockets.

    eg. #define PROFILE 1 in SoakTest.cpp

    Disable sleep in this mode.

    Make sure SoakTest is working properly with -NDEBUG. 

    My guess is is that it was just running too fast to have the IO function.

    The profile mode above should help with this. Verify messages are received.

    Verified. It was just timing with the sleep.

    Disable printf in this mode.

    Look and see what is the most expensive now.

    Now the split looks like this:

        71.4% - Connection::WritePacket
         9.9% - Connection::ReadPacket
         2.7% - ConnectionPacket::~ConnectionPacket
         2.4% - shared_ptr
         0.5% - ReliableMessageChannel::SendBlock
         0.3% - ReliableMessageChannel::SendMessage
         0.1% - ReliableMessageChannel::CanSendMessage

    There is definitely something going wrong inside Connection::WritePacket

    Looks like there is an expensive vector allocation going on inside there?

    I'll bet this is the fragment alloc on send, maybe because it is clearing the data to zero?

    Fixed the clean to zero but no significant change. I think it's the alloc itself.

    I think what is going on is that there are just significantly more packets written
    that read due to the simulator dropping packets.

    Changed the code so that all written packets are read.

    Lets see what is expensive now:

        31.3% - Connection::ReadPacket
        24.0% - Connection::WritePacket
        20.7% - shared_ptr<vector<uint8_t>> allocator (!!!)
         4.8% - ReliableMessageChannel::SendBlock
         3.0% - ReliableMessageChannel::SendMessage
         1.2% - ConnectionPacket::~ConnectionPacket
         1.1% - ReliableMessageChannel::CanSendMessage

    Bottom line.

        1. Read packet seems a bit slower than write packet. Why is this?
        2. I need to use a custom allocator
        3. shared_ptr is stupid. I should not be using it


Tuesday Morning, June 10th, 2014:
---------------------------------

    Inside Connection::WritePacket almost all the cost (18% of 19% total)
    is inside ReliableMessageChannel::GetData

    Start here!    

    Instead of walking the send queue to find the oldest message id not acked,
    cache this value, and on each update, walk left to right to skip past acked
    messages in the send queue to get the new value.

    This has reduced the cost of "GetData" considerably.

    Now the largest cost is serialize. It makes sense to do the serialization fixes first.

    Bring back templated serialize functions.

    Yes, network object should have virtual SerializeRead( ReadStream & stream )
    and SerializeWrite functions, BUT... these can be implemented via templated 
    serialize fn in the class.

    I think this is the best of both worlds, and avoids any overhead in the
    serialize due to extra branching.

    Reworked "Stream" into separate "ReadStream" and "WriteStream"

    Reworked template functions to template <typename Stream>, so we can still get
    Stream::IsReading, and Stream::IsWriting (easy to read and scan)

    In base object, there are now two virtual methods: SerializeRead and SerializeWrite

    Pass over BSDSockets.cpp and get it working with separate read and write streams.

    Converted BSD sockets over. Had to do a bit of shennanigans with typedef WriteStream Stream
    for the macros. Don't know how else I could do this, vs. crafting a templated serialize fn.

    Convert Connection.h and TestConnection.cpp

    Converted ReliableMessageChannel.h

    Convert soak test.

    Profile results:

        62.6% - sleep
        24.5% - BSDSockets::SendPackets
         9.1% - BSDSockets::ReceivePackets
         1.7% - Connection::WritePacket
         0.4% - shared_ptr
         0.4% - Connection::ReadPacket
         0.2% - more shared_ptr
         
    Now try with -DNDEBUG

    Not sure if I can trust the result, but:

        70.9% - sleep
        23.8% - BSDSockets::SendPacket
         2.3% - Connection::WritePacket
         1.7% - BSDSockets::ReceivePackets

    Inside BSDSockets::SendPacket:

        22% (of 23.8%) is in the sendto... looks like we are IOBound in this test.
    
    Convert client/server.

    Everything works. Time for a break.


Early Tuesday Morning, June 10th, 2014:
---------------------------------------

    Profile SoakTest.

    Expensive parts are:

        ConnectionPacket::Serialize (read and write -- consider templated read/write stream types)
        sendto and recvfrom system calls (should be on another thread!)
        ReliableMessageChannel::GetData --> SlidingWindow::Find (looking for oldest unacked message? cache it!)
        ReliableMessageChannel::GetData --> sequence_less_than

    Misc bits that should be fixed:

        memset 0 for sockaddr_in (removed)
        shared_ptr overhead (what is the alternative?)
        std::string widening, allocations, format_string etc. (disable logging!)

    Switching to an optimized build and it looks like this:

        58%  - sleep
        13%  - BSDSockets::ReceivePackets
        7.4% - Connection::WritePacket
        5.7% - format_string (remove this! clearly it is awful...)
        4.8% - basic_ostream (printf is looking really good right about now...)
        4.3% - BSDSockets::SendPackets
        1.8% - basic_ostream again
        1.7% - shared_ptr (alternatives?)
        0.5% - Connection::ReadPacket
        0.2% - basic_ostream again
        0.2% - more shared_ptr stuff.

    From this it is clear: my goal is to get the code as close as possible 
    to having all the time spent in the sleep, or in functions like sendto
    and recvfrom over which I have no control.

    All other functions are *fair game*.

    It is clear that:

        a) I should cache the oldest message to avoid SlidingWindow::Find per-get data
        b) I should optimize the bitpacker, most likely splitting read and write via templates
        c) format_string should die in a fire
        d) printf is better than cout
        e) shared_ptr is slow. i need an alternative
        f) allocating stuff is slow. need a custom allocator

    After removing printf calls:

        57.8% - sleep
        19.3% - Connection::WritePacket
         9.4% - BSDSockets::ReceivePackets
         7.7% - BSDSockets::SendPackets
         2.5% - shared_ptr
         0.7% - Connection::ReadPacket
         0.2% - shared_ptr
         0.1% - ReliableMessageChannel::ReceiveMessage
         0.1% - ReliableMessageChannel::CanSendMessage


Monday Evening, June 9th, 2014:
-------------------------------

    Add optimized "serialize_bytes" which aligns and then just memcpys in/out.

    Add this to stream.

    Then add to BitReader, BitWriter.

    Hook this up to be used in all cases where blocks are serialized.

    Now that it is functional, make it faster going through memcpy in/out.

    It should be a *lot* faster than the bitpacked version, for large blocks!

    It's quite tricky! Have to split into head/body/tail to work with
    the bitpacker word based operation, and this is not trivial.

    Got it working with "ReadBits(8)" in the body loop, but need
    some more work to get it working with memcpy, eg. have to
    remember how I did the "ReadBits" implementation w. scratch.

    Got it working. Needed to clean up the word index a bit.

    Once this is working, apply the similar memcpy optimization
    to write bytes, with head/body/tail.

    Seems to work fine. Running soak test to verify.

    Can optimize messages id serialization quite a bit, by encoding next message id 
    relative to the previous one, eg. +1, +5, up to n where n would actually be small, 
    eg. half sliding window size.

    This is a big optimization.

    Common case would probably be a bunch of messages in order too, eg.
    one bit = +1, repeated n times.

    What would also be awesome about this is that this would convert
    the message ids into something that is more likely to be *constant*
    or at least, would often have repeating patterns from packet to packet.

    When doing this, make sure that the message types are followed
    by the message ids in a separate array, vs. being interleaved
    with message data.

    This would help out the dictionary based LZ compressor quite a bit,
    vs. the current bunch of message ids that change from packet to packet.

    Add a function to serialize an integer relative to another integer.

    Use this to serialize the message ids.

    Holy shit it works!

    Fixed an issue with message id wraparound past 16bits with relative int encode.



Sunday Evening, June 8th, 2014:
-------------------------------

    Add "ReadAlign" and "WriteAlign" methods to bitpacker classes.

    Added serialize check, eg. "Check" which takes a magic value.

    Add an "Align" method to stream.

    Add a "Check" method to stream, take in a magic number uint32_t

    Verify it works, and add asserts on read back to verify remainder.

    Extend the connection header to read/write align after packet header.

    Change the connection to write bits of channels which have data in
    one block, vs. one bit, then channel data etc. Will compress better
    with something like oodle.

    Add alignment in front of each channel data serialized.    

    Need a function to calculate align bits at given stream point.

    eg. GetAlignBits for bitpacker read and write.

    Add config inside reliable channel config to add alignment
    per-message, eg. align on message id, and align before message
    contents, for block serialization, add align before block data
    also.

    Extend bit of code to conservatively estimate including
    padding, eg. add n bytes overhead (seems 4 bytes) max,
    plus up to an extra byte per-message should do it.

    Align seems to be working in unit tests.

    Now try soak test with align... seems to be working!

    My guess is that the align check is highly conservative,
    eg. most time we have room for a few more messages but
    just don't bother, so it's no big deal.

    Implement reset for reliable message channel, needed for connection and client/server.

    Clean up counter names, remove that bad "PacketReadFailures" counter
    and replace it with something that makes more sense, eg: PacketsDiscarded

    Clean up the error handling from "ProcessData".

    Basically, if it returns false it means discard the packet, but keep 
    processing the rest of the channels, basically -- don't ack this.

    I've managed to break the reliable message channel somehow. FML

    Fixed. It was the case where I just have to skip and do nothing,
    and I had incorrectly switched it to return false there.

    Fixed missing alignment after packet type inside BSDSockets.h

    Can optimize the packet header by encoding the acked relative
    to the sequence, eg. sequence - n = ack #, where n is typically
    in range [1,32] or so.

    Also, shuffled packet header so that infrequently changing values
    are grouped together, and frequently changing values are separate.

    This should help the dictionary based compressor do a better job.


Sunday Morning, June 8th, 2014:
-------------------------------

    It's really annoying having to process the disconnect packet
    and having client guid and server guid stashed in the per-state
    data. Maybe it they should be global data instead in the client?

    Fixed by moving clientGuid, serverGuid and address out of the state data.

    Removed start time.

    Moved out accumulator.

    Clean up the timeout by having a unified "timed out" state,
    but with the extra error specifying the state the client
    was in when they timed out. Works nicely!

    Update unit tests to be fixed according to this new setup.

    Client side unit tests are broken now. Find out why.

    Update timeout was not getting called.

    Also, resolver was taking a garbage hostname and returning
    a valid IPv4 address. Fixed by specifying IPv6 only in the
    DNS lookup. Seems strange though.

    Unify the timeout processing and the accumulator fn. for sends
    when a packet should be sent, call a function that checks current
    state and decides to send or not.

    Finished refactoring Client.h 

    It's good enough. Time to move on.

    There is an issue now with sending packets in test_client_resolve_hostname_success

    What is going on?

    It's because no port is specified in the hostname and the address has port 0.

    Fixed by adding a "defaultServerPort" to config and setting the port to that
    if the resolved address has port zero.

    I also need to extend hostname parsing to support "host:port" format (optional)

    But I can do this later, it's not blocking me.

    Added a test for connected denied server full

    Added a test that covers client and server side timeouts

    Decide what should happen if the same address (client) tries
    to connect while still being connected (by address only).

    I think it should be denied "already connected", otherwise
    it's a bit too easy to DOS a server, eg. just continually
    add connections.

    Implement server rejecting client if they are already connected

    Added new reason for denying connection request:

        CONNECTION_REQUEST_DENIED_AlreadyConnected

    Add a test for this case. Passes!

    Add a test for client reconnect.

    In this case you must disconnect the client fully before reconnecting,
    server side "DisconnectClient" makes the most sense here.

    I don't really want to do the block sending right now.

    Do some small cleanup tasks.

    Hostname parsing needs to handle port

    eg. "localhost:10000" should resolve to port 10000 on the hostname that gets looked up.

    Remove the integration of the resolver with the network interface.

    Remove resolver from BSDSockets interface.

    Move it to another class, "ResolveWrapper" and update tests to use this.

    PacketsSent and PacketsReceived counters were broken in BSD sockets.

    Fixed them and added a unit test to check they are working.

    Appears to be something wrong with acks.

    Fixed it, had a few logic errors in there that showed up 
    only as strange behavior in soak test, eg. long periods
    with no acks, an assert firing randomly etc.


Saturday Morning, June 7th, 2014:
---------------------------------

    Verify connection is *functional*. eg. exchange a few client
    and server events back and forth.

    Messages are working. Success!

    Add a test for server side disconnect

    Added method "DisconnectClient" to server.

    Need to implement disconnect client, first implementation
    could just send a disconnect packet to client then reset
    the client slot.

    Add client code to handle the disconnect packet and 
    go into disconnected state. Set error to indicate
    client was disconnected.

    Verify test passes. Yep.


Friday Morning, June 6th, 2014:
-------------------------------

    Implement the case where client and server blocks AREN'T specified.

    eg. make them optional. In this case, skip ahead to states that say
    go to the next state, eg. server skips immediately to "send client block"
    state and then client immediately on getting that packet goes into
    "ready for connection" state.

    On the server, if the block is nullptr, go immediately into
    "requesting client data block" state.

    Cleaned up packets. It's now "server data" and "client data".

    Cleaned up states. Added "requesting client data" and go to it
    immediately after challenge response, if the server data is nullptr.    

    Implement the server sending "request client data" packets
    while in the requesting client data state.

    Cleaned up the client states, and consolidated timeouts
    and send rates into "connecting" and "connected" timeouts.

    On the client, when we get the "request client data" go either
    into the "sending client data" state *or* if the client data is
    nullptr, just go directly into "ready for connection" state.

    On the client while in "ready for connection" state just send
    out "ready for connection" packets to the server.

    On the server, if a "ready for connection" packet is received
    and the server is in the "requesting client data", just go
    directly to connected state.

    On the server in the connected state, send connection packets

    On client, when a connection packet is received while in the
    "ready for connection" state go directly to connected state.

    Verify client and server are in connected state.


Thursday Evening, June 5th, 2014:
---------------------------------

    Extend server code to listen to challenge response packet.

    Clean packet processing on server up a bit, there is now
    a "ProcessX" function for each packet type.

    When challenge response is received, go into sending data block state.

    Add test to verify this happens. Keep it separate from previous test.


Thursday Morning, June 5th, 2014:
---------------------------------

    There is a crash in the server creation or update. What is going on?

    Verified it is in the update, not the create.

    It appears to be in the address comparison on packets.

    I don't really understand how this can be happening?!

    It seems to be caused by a fall through that should have
    broken out of the packet receive on m_open == false.

    Still not sure why this would crash though?!

    Added a new test with the server open. 

    It crashes again, my guess is that there is something broken
    in the client data array on the server?

    Found it. Bad condition in if statement.

    Add code inside UpdateSendingChallenge to construct a challenge packet
    and send it out at a specific rate, eg. add accumulator to the client
    data struct.

    Added test to verify client progresses to sending challenge response state.


Tuesday Morning, June 3rd, 2014:
--------------------------------

    Add function "FindClientIndex" which returns a client index, or -1
    if no matching client is found.

    Add a new function "FindFreeClientSlot", which returns client index
    of a free slot, or -1 if no free slot is available (server full).

    Add code to take a client request packet and search for client index
    matching address and client guid.

    If no matching client slot is found, search for a free slot.

    If a matching client slot is found, don't do anything. Ignore packets.

    If a free slot is available, set up that client slot with the client
    guid, address, generate a new server guid for the slot.

    If no free slot is available, reply with connection denied server full.

    Add function "FindClientIndex" which returns a client index, or -1
    if no matching client is found.

    Add a new function "FindFreeClientSlot", which returns client index
    of a free slot, or -1 if no free slot is available (server full).

    Add code to take a client request packet and search for client index
    matching address and client guid.

    If no matching client slot is found, search for a free slot.

    If a matching client slot is found, don't do anything. Ignore packets.

    If a free slot is available, set up that client slot with the client
    guid, address, generate a new server guid for the slot.

    If no free slot is available, reply with connection denied server full.

    Added timeout config and code inside server to time out client slot
    and free it up for future connects if no packets received.


Monday Evening, June 2nd, 2014:
-------------------------------

    Actually implement the server side per-client.

    Each client needs their own connection, their own current state.

    Define the set of server client states.

    Add server full to the list of connection denied reasons.

    Implement the code that takes the client connect request, and if not closed
    looks for a matching client with address and client guid, if that client
    is already setup, ignore the packet.

    The number of clients is small. Therefore there is nothing wrong with
    doing a simple linear search on connection request to find a matching
    client index.

    Sketching out how to do this, then going to bed.


Monday Morning, June 2nd, 2014:
-------------------------------

    Having some problems with multiple interfaces.

    The packet is sent, but the second interface isn't receiving it.

    Seems strange...

    Moved over to a couple of unit tests.

    I'm seeing the same behavior in IPv4 and IPv6, so it's not something
    that is specific to my IPv6 usage in the test (::1)

    What is going on? My guess is maybe it's screwing up the port number somehow?

    Fixed it! I was checking errno, even when there was no error, this was
    picking up the EAGAIN from the other socket, dropping the packet! :)

    Connection denied unit test passes. Server is getting packet, replying
    with a connection denied and the client is properly handling this.


Sunday Evening, June 1st, 2014:
-------------------------------

    Need to solve issue of having multiple connections, each with a shared
    channel configuration, and needing to share the same packet factory.

    Basically, it seems we need some concept of a channel configuration
    object, eg. something that can say how many channels we have, and knows
    how to create channel data on request.

    Ideally, this channel configuration should be passed in to the connection
    so it can be reused across multiple connections.

    If this channel configuration was the thing passed in to the factory,
    my problems with the server packet factory would be solved.

    After a lot of work...

    I have created a new class "ChannelStructure". 

    It answers the question, how many channels are there?

    As well as providing a mechanism to create a channel,
    and a way to create channel data on request.

    On integrating it with unit tests, it becaume clear that it also
    needs to manage the *configuration* for channels, as these channel
    must be created knowing their configuration, eg. reliable message
    channel needs configuration, including a message factory, and
    reliable message channel data needs config data too.

    It seems to work well, and it's definitely what we need -- a way
    for the user to specify the exact channel structure, and the connection
    to not really know or care about it, just to do the work given the channel
    structure passed in to it.

    It works well.

    Need to pass over remaining reliable message channel tests and convert
    to use new channel structure.

    Done.

    Build all unit tests and make sure they all compile and pass.

    Client server test needs some work... done.

    Updated server to use the channel structure in config.

    Soak test needs some work. Converted.

    Something is wrong with soak test.

    I get this assertion:

        Assertion failed: (m_bitsWritten + bits <= m_numBits), function WriteBits, file ../include/BitPacker.h, line 34.

    Which is strange, because I should have made no change at all to the behavior of the soak test.

    Basically says I am writing past the end of stream.

    What is going on?!

    The channel config may have changed. I'll bet the channel budget now exceeds the max packet size.

    Yep. That was it! Fixed.


Sunday Afternoon, June 1st, 2014:
---------------------------------

    Add connection deny packet. Inside, have clientGuid as well as 
    one uint32_t explaining the reason for deny. 

    First reason should be server is closed. 

    Second reason should be server is full.

    User should be able to determine, via OnClientConnectRequest whether
    the connection should go ahead, and what the reason is for denying.

    Implement the server and add a way to enable/disable the server from
    accepting connections, eg. open/close. Server should initially be open
    but if you want it closed, call close before the first update pump.

    Added uint32_t extended error to client. This will be useful for 
    storing the connection denied reason.

    Implement the code on the client to handle connection denied and
    transition into "connection denied" error with reason in error ex.

    Seems that we don't really need to pass a packet factory in to the config
    for client and server.

    In fact, on the server we actually need a factory per-client connection
    because it links the packets to the connection class they belong to.

    Refactor the client and server configs to not require passing in packet
    factory. Let the server and client create their own packet factories instead.

    Nope. It's worse!

    I have a real dilemma now.

    The socket interface uses the factory to link up the connection packet
    to the connection interface, which handles config and so on, but this
    is specific to each connection.

    It seems that on the server I'm going to need to maintain a separate
    connection interface used by all clients, vs. one per-connection.

    In hindsight it seems that the connection interface idea per-connection
    is a mistake.

    It seems that I need to separate the channel configuration from the 
    connection itself, as the channel configuration would be *identical*
    across all connections, and in fact, I'd want to know about it
    *before* any connection has been established.


Sunday Morning, June 1st, 2014:
-------------------------------

    Add the connection request packet type to ClientServerPackets.h

    Added connection challenge, and challenge response packets.

    Each packet has a protocolId uint64_t, as well as a client guid uint64_t
    and a server guid uint64_t. This should provide enough safety that the
    packets belong to the same client, with same protocol and that connection
    can be established.

    Once connection is established, the 64 bit protocolId is passed along
    with each connection packet. This is overkill if travelling over a
    transport that already establishes connection, eg. UDPP2P on PSN,
    but for a raw UDP transport this is appropriate.

    I can repurpose this 64bits to support packet signing data
    and encryption payload, to avoid man in the middle attacks later.

    Added very rough code to generate 64bit guid. This needs to be revisited.

    Make sure the client generates a guid at the start of connect by address.

    This guid should be passed in to the server on connect request,
    and this guid is carried along from the client all the way to
    successful connect (to avoid mismatching client packets from
    a previous connect attempt...)

    Implemented state update for sending connection request.

    Implemented timeout if in sending connection request too long.

    Added unit test to verify sending connection request timeout.

    Actually start sending the connection request packets.

    Add a config for the send rate while in this state. 10 times per-second default.

    Now getting an error in sendto when actually sending the packets. Why?!

    Fixed. I thought it was defaulting to IPv6 but actually was defaulting to 4.

    Write the client side code to receive incoming packets.

    Knowing that only the client may receive packets on the interface
    leads me to make the client responsible for updating the interface
    as well, and updating the resolver.

    Added code to process packets, if the packet is connection challenge
    and the client is in correct state, and all data matches, client will
    transition to the sending challenge response state.

    This is probably as far as I can go on the client until I implement server side.


Saturday, May 31st, 2014:
-------------------------

    First, start by defining the client and it's state machine, behavior etc.

    Client will have a single connection, a network interface it uses to
    send packets across.

    Client/server will have to have their own packet enum, packet factory
    implementation, as they need a bunch of their own packets for comms.

    Add resolver and interface to client and server configs.

    Add ClientServerPackets.h

    Added tests/TestClientServer.cpp (no point testing in isolation!)

    Sketched out the set of tests I want.

    Designed the client state and the set of errors.

    Started work on basic tests, eg. initial state.

    Added lots of nice accessors, IsConnecting, HasError, IsConnected
    IsDisconnected and so on. There functions are shorthand, but it makes
    the code quite a bit clearer vs. checking state/error enums all the time.

    Implemented internal functions for connect/disconnect, and error setter,
    clear and so on.

    If hostname is an address string, use that and connect by address instead.

    Implemented hostname lookup via resolver. Added per-state data so the
    state machine remains really clean. Added fn. to clear all state data,
    and made sure it gets called on disconnect.

    Should now have code that actually performs the resolve, waits for
    succeed or fail result, as well as having timeout specified in config
    at which point it gives up (important for every state!)

    Add a unit test to verify the timeout works, trick would be to not
    update the resolver. No sleep required either.

    Add test to verify the hostname resolve works, eg. use "localhost"

    Added code to transition into "sending connection request" state
    inside connect by address fn.

    Enough work for today. Time to take a break.


Friday Morning, May 30th, 2014:
-------------------------------

    Implement ack test. Randomly drop packets and verify that the invariant holds
    that the acked packets must have been received, and that not received packets
    must not be acked.

    Note that it's possible for acks to be lost, so some received packets will not 
    be acked.

    Need a way to determine when a sequence has been acked.

    Probably the simplest way is to implement a dummy channel that hooks in
    to the connection and has "ProcessAck" called on it.

    Added test, works fine.

    Make sure all tests call srand( time( NULL ) ) at the beginning,
    so they have actual random numbers, vs. the same sequence every
    time they are run.

    Add a reliable message channel test that mixes messages, large and small blocks.


Thursday Night, May 29th, 2014:
-------------------------------

    Add latency and out of order packets to reliable message channel tests.

    Found a breakage in the large block test!

    Wow. Fixed it to throw an assert, and it seems that it is now stalled out.

    Soak test hangs too if you leave it long enough. Seems to hang right after
    a long block is received, going back to bitpacked messages / small blocks.

    I have a 100% quick repro with the large block test, so debugging there first.

    First see what it is hanging up on, log the packet discards... Nope.

    It seems to be another ack related issue.

    My theory is that the sender has missed some acks, so although the receiver
    has definitely received the packet (and has advanced on), the sender does not
    know that he is finished, and can stop sending this block.

    I think the fix is for the receiver to only discard the packet if it is a large
    block id that is more recent than the current message id. Older block ids need
    to get acks back to resolve this situation.

    Yep. That did it!


Thursday Morning, May 29th, 2014:
---------------------------------

    It would be nice if the soak test would cycle in-out of really bad conditions

    eg. cycle between 0 packet loss and 100% packet loss.

    Cycle between no latency and 10 seconds latency.

    Cycle between no jitter and 1 second jitter.

    The goal of the stress test is to test the worst possible conditions.

    This means that the test actually has to exercise these worst case
    conditions, eg. 10 seconds with no packets getting through, does it
    recover?

    It would be nice if the simulator itself had the ability to cycle between
    different states with probabilities, eg. create a bunch of "state" structs
    with settings for latency, jitter, packet loss -- and assign a probability
    to each (which gets normalized), then pick a new state randomly with 1 in n
    chance (n in config) every update.

    It's great to put this in the simulator, vs. having it in the SoakTest,
    because this same logic can be used anywhere the simulator is used.

    First step, split out the simulator into NetworkSimulatorConfig (create time)
    and NetworkSimulatorState (dynamic).

    Added "AddState" function. This way you can add states to the sim so that
    it oscillates between them with equal probability. I don't really think
    the weighting by priority is worth the effort. This is good enough.

    Verified that the state is adjusting every n frames.

    Soak test works fine.


Wednesday Morning, May 28th 2014
--------------------------------

    Add "NetworkSimulator.h" to delay packets and deliver them
    after a fixed amount of delay, with +/- jitter in seconds.

    Made network simulator implement the network interface.

    Seems to make the most sense.

    Add latency and out of order packets to soak test.

    Seems mostly fine, but when I increased the amount of packet
    loss and latency, it seems I have broken it.

    Interestingly, just setting latency to 1 second is enough to break it.

    Also, it still breaks when I only send messages. Large/small blocks not required.

    It breaks *precisely* at 512, which is the receive queue size.

    What is going wrong?

    It seems that all messages are being discarded on receive because they are "old"

    My guess is that somehow the receiver is running ahead, eg. has the sliding
    window advanced forward, further than will allow the non-received messages
    to ever be received.

    It seems like "ProcessAck" is not getting called.

    Why is this?

    Acks are also super weird, eg. for a long time:

        65535 - 0

    And then it ends with persistent:

        n - 55555555

    Which is really strange as well. Why 55555555?

    In binary this is 010101010101...

    Why are acks coming through for odd packets?

    Some seriously fucked up stuff here.

    Some clues here:

    ack patterns are consistently 0101010 etc.

    This persists even if:

    a) bsd sockets are removed
    b) simulator is removed

    This is very strange. Is something wrong with acks or is send sequence getting
    bumped by two somehow? eg. ++, ++?

    Yes. It looks like sequences are going up by two somewhere.

    This *may* be a separate issue to why the event queue stalls out at 1 sec latency.

    Yes, the sliding window change I made was adjusting the sequence ahead at twice
    the rate, eg. 1, 3, 5, 7 etc...

    This was also causing the broken acks, eg. the sent packet data did not match
    the actual packet sequence, thus lookup for acks was broken, acking the wrong
    messages in the send queue and causing the messages to stall out.

    Fixed!

    Reintroduced mix of bitpacked messages, small blocks and large blocks into
    soak test. Everything is working fine again.


Monday May 25th, 2014 (Memorial Day)
------------------------------------

    Add some rare large blocks to the soak test. 

    Working fine.

    Bumped up soak test so it's really going to be pushing the limits of CPU,
    eg. larger packets, large blocks, increased message size etc.

    During soak test it would be nice to see received x/y fragments of large block.
    This way there is continuous logging and it is a nice steady stream, vs. the current
    large hitch on each block which is occuring.

    Added GetSendLargeBlockStatus and GetReceiveLargeBlockStatus to return 
    structs providing information about current send/receive large block.

    Used these to display x/y fragments while receiving a large block.

    Extend soak test to actually validate contents of blocks properly,
    eg. make each byte a modulus of byte index in block.

    Make the small and large block data more distinct, eg. each byte
    a unique modulus of byte index and sequence, and verify each byte
    is correct on receive.

    Unit test should have some bit pattern for message bits,
    so they can be verified on the way back.

    Solved by adding a magic 32bit value after the bits.


Sunday Evening May 24th, 2014:
------------------------------

    Things to do to get the large block codepath working:

        1. Implement code to include a fragment in packet data          <-- DONE

        2. Implement code to fill the sent packet entry                 <-- DONE
           for fragments, so they can be acked. Need both
           message id for block, and fragment id.

        3. Implement code to process ack, lookup block by id            <-- DONE
           id and fragment, and mark that fragment as acked,
           and once all fragments are acked, go to next msg
           in send queue.

        4. Implement code to serialize read/write fragment data.        <-- DONE
           It's OK to copy the fragment data from the packet to/from
           the actual block itself, this is no problem given small
           fragment sizes, and it makes the code implementation
           much easier (actual block remains private to the channel)

    Hey, I have to store the actual block size inside the serialize data
    otherwise the receiver has no idea of the actual size. I think just
    sent a 32bit block size with each fragment, that way any fragment
    arriving first will prime it up. Done.

        5. Implement code to take that serialize fragment data
           and copy it in to the block itself on receive,
           track received vs. not received fragments, and
           queue the actual block up in receive queue once
           all fragments have been received.

    On receive side of block I need:

        - active true/false whether we are receiving a block right now
        - blockId that is currently being received
        - size of the block that is being received
        - number of fragments in block that is being received
        - number of received fragments in block so far

    When a fragment comes in for a block, first check if we are already
    receiving that block.

    If not receiving, check that we have received all messages ids 
    prior to the block message id, if not, then discard packet
    and bump a counter. This should "never happen".

    If already receiving, verify the fragment id is in bounds, 
    and then check that the block size still matches.

    If we are receiving a block, discard any packet that arrives
    containing a different block, or non-block messages. This should
    be an exception and raise a counter because it should "never happen".

    Had to adjust the sliding window so that "GetSequence" returns
    the next expected sequence to insert, vs. the current one -- which
    was undefined and not giving me the result I wanted for my "is this
    large block id the expected next mesasge to be received" check.

    Had to adjust various tests, as well as GenerateAckBits to work
    with this new scheme. But it makes more sense and the code is
    clearer post this adjustment.

    Seems there is some data corruption on receive. A byte at a weird
    index (1985?) is zero in the second received block. Why is this?!

    Gotta debug why the block is getting corrupted.

    Seems like it is, at least for the first block being sent, off by 64

    eg. 1024 - 960 = 64

    My guess is an off by one on the last fragment?

    Passes now for first block, but second block which is the first
    one that has a non-even multiple of fragments, has a problem with
    the last 64 bytes

    eg. 2049 - 1985 = 64

    My guess is that I'm off by one in the case of the block with remainder?

    Yep. Off by one. Was not dividing as float inside ceil.

    Now I'm getting an assert in the large block test, under packet loss:

    Assertion failed: (data.blockId == expectedBlockId), function ProcessData, file ../include/ReliableMessageChannel.h, line 589.

    Is this a valid assertion? It seems to indicate that a block is coming in
    but with an unexpected message id. But can this possibly happen, legitimately?

    It would seem it could happen if a block goes incorrectly to start sending
    the next block, before the current block is fully acked. Is this happening?

    It seems that under high packet loss the sender can still be thinking
    that he is on block 12, when the receiver is ready for block 13.

    I'm guessing this is just a delayed reaction, eg. some acks that don't
    get through. The sender recovers and works just fine, but exception throwing
    in packet processing, it's not an exceptional thing -- can totally happen.

    These aren't conditions that are assert worthy. They can happen in the real world.

    Are there other conditions I thought which were imposible, but can actually
    happen under high packet loss? Probably!


Sunday May 25th, 2014:
----------------------

    User should be able to specify large block fragment size in config,
    and maximum block size (so we can derive the max # of fragments)

    I think having up to 16 bits fragment index is plenty. If you want
    large blocks, you'd naturally want larger packets and larger fragments,
    otherwise it's just a silly exercise to think to try to send a gig
    file through 32 bytes at a time =p

    Need to allocate a structure up to the maximum fragment size, one
    on send and other on receive side, for per-fragment data.

    The structure should be extremely well packed to efficiently support
    large blocks, and be cache coherent.

    The actual block data itself should be dynamically allocated
    and passed around via shared_ptr. Blocks are large. No copying!

    The per-sent packet sliding window needs a new entry showing if
    this packet had a block fragment, or if it contains messages.

    Send queue entry needs a "largeBlock" bit for easy processing
    of the message /small block mode, it needs to go only up to the
    next large block.

    Added code to break on nullptr send queue entry, because that should mean
    STOP, there is no entry for this message id. Need to verify with soak test.

    Yep. Works fine.

    Added split for "large block" vs. "message and small block" mode inside
    get data. Need to implement the large block mode codepath. For the moment
    it just returns nullptr data (stalls out)


Saturday May 24th, 2014:
------------------------

    Shrink the message size in soak test now that we have bitpacker

    Add overflow checking for bitpacker read and write.

    Add exception on overflow for serialize read. This is the important one,
    because this data can be coming in from an untrusted source.

    When writing a packet, it is expected that programmer error will be
    the only source of writing past the end of the buffer, thus the checking
    is performed via asserts. No need for exceptions in this case.

    Implement code to measure messages in send queue.

    Ensure that this measurement is done only once as it is expensive.

    Cache the message overhead # of bits in ctor so it doesn't have to be recalculated.

    Shrink the size of the packet in soak test so we can't fit all messages in the packet.

    Also jack up the max # of messages per-packet to 64, so small messages can get packed in.

    Implement code to count the # of bits for messages when searching for messages to include
    in the send data.

    Make sure there is code that gives up after a small # of bits remaining, so the code
    doesn't walk through the remaining send queue entries unnecessarily when no message
    can possibly/reasonably fit.

    Make sure we never go over max packet size in soak test

    Make sure the reliable message test passes


Monday May 19th, 2014:
----------------------

    Extend message channel to support *small* block messages

    This can be implemented just by checking the block size, and implementing a serialize
    that just serializes the bytes to the stream via the bitpacker.

    Extend MessageChannel interface to support sending a block (vector ...) directly.

    Assert if message is too large to include (add small block # of bytes in config)

    Extend soak test to include non-block messages with random # of bits

    Optimize block serialization to serialize by words (4 bytes at a time)
    with tail serialized one byte at a time. Less calls to serialize,
    and more efficient journalling (although, journalling should be,
    if implemented, coded to be aware of the block, eg. serialized block
    n bytes long, should be the journal entry...)

    Write a generic method, serialize_block to provide block serialization
    outside of the block message. This is generally useful functionality.

    Extend the soak test to include block messages of random # of bytes

    Implement the bitpacker.

    Keep bitpacker separate from the stream class for maximum flexbility.

    Bitpacker should work with raw pointers, not vectors.

    Bitpacker should not use or throw exceptions. Leave this to stream layer.

    Split into separate reader and writers. I don't want unified read/write.

    Implemented bit writer.

    Added unit test to make sure # of bits are written, bits available correct etc.

    Using htonl for endian conversion. Later on my want to use my own routine,
    otherwise this will require me to include headers just to get htonl.

    Added some code that will let me detect endian at runtime, without
    needing to pass in #define BIG_ENDIAN or LITTLE_ENDIAN.

    Is there a better way to do this? eg. at compile time vs. runtime?

    Apparently there is not.

    Implement bit reader.

    Fixed bug on write that set all bits to 1. Was incorrectly trying
    to clear high bits on write bits, ended up setting all bits to 1.

    Got it apparently working but I'm getting some puzzling logs.

    The read and write words are not matching.

    This is puzzling. How can these logs be wrong, when the values
    being read back from the bit reader are correct?!

    Dumbass. It was just endian flip.

    Hook bitpacker up to the net stream.

    Make sure stream tests pass.

    Make sure reliable message channel test passes.

    Make sure soak test still works.

    Implement serialize_bits( stream, int, num_bits ) -- this is very useful

    Converted serialize_bits and serialize_int to macros so they work with 
    bitfields. Can't pass a bitfield in by reference. Macros work.

    Pass over existing serialize_int and switch over to serialize bits where appropriate

    Add serialize_bool

    Passed over code using serialize_int and switched to bool where in range [0,1]

    Added optimization to ack_bits. If it is perfect (0xFFFFFFFF), serialize one bit.

    There is a problem with TestReliableMessageChannel, seems that some data is messed up (message id?)

    Fixed. Write stream was missing a flush.

    BSDSocketsInterface.h needs conversion (it uses serialization internally)

    Verified BSD sockets test passes

    Convert the SoakTest.cpp over

    Verify it still runs.

    Time for bed!


Sunday May 18th, 2014:
----------------------

    Extend soak test to send and receive messages

    Remove logs except send and receive message n. 

    Make sure all asserts were being checked. #ifndef NDEBUG

    Fixed bug with adding messages to packet. Message id had modulus
    applied so after sliding window size, no messages from send queue
    where being put in packet data.

    Some work to ensure that send -> receive messages is the minimal
    number of frames, ideally messages should be received at the end
    of the same frame they were sent, for loopback.

    Works fine without packet loss, but with packet loss it stalls out

    If I disable the last send time check for resending a message,
    it works again. It seems there is something wrong with resend
    logic?

    There was an off by one when the send queue was full,
    such that message 0 was never being resent, once the send
    queue filled up.

    Also, in the case where the receive queue is smaller than
    the send queue (common), the send side was sending early
    events to the receiver, which is counterproductive.

    Fixed by only sending events up to the size of the receive
    queue from the oldest message in send queue, nothing more
    recent than this is ever sent.

    Structure source code, eg. include / src / tests

    Update C++11 build system to look up for ../includes


Saturday May 17th, 2014:
------------------------

    Sketched out soak test.

    For some reason packets aren't being received from the interface

    Fixed. I was not passing in a resolver and the update was returning
    early if resolver was null (not running send packets)

    Split out resolver update into "UpdateResolver" so the other 
    update part is not accidentally being skipped with null resolver.

    When connection packet is deserialized it asserts because it doesn't
    get registered with its connection interface.

    I think I should return the connection packet to require a connection
    interface being passed in, and a custom type vs. hardcoded to some
    value, this way the factory is responsible for its creation!

    Solution is probably to have a custom factory that does this registration.

    Rename "factory" in configs to "packetFactory" or "messageFactory"
    to make it clear. There will not always be one factory in context.


Thursday May 15th, 2014:
------------------------

    Fixed up connection packet so it conforms to standard factory interface.

    Fixed TestConnection.cpp so it works again.


Tuesday May 13th, 2014:
-----------------------

    Restructure code into various headers so I can have multiple programs.

        Common.h
        Stream.h
        Network.h
        Connection.h
        MessageChannel.h

    Split up unit tests into separate cpp per-aspect, eg. TestMessageChannel.cpp

    Split up the code so that each section has minimum dependencies.

    Problem. I would really like packet to be independent of network.

    Solution: Move address out of network.h

    Moved Packet.h out of Network.h

    Moved Channel.h out of Connection.h

    Moved Message.h out of MessageChannel.h

    Move various X::Configs out into their own XConfig structs. More flexible.

    Create test harnesses for each module:

    eg: 

        TestCommon.cpp
        TestStream.cpp
        TestNetwork.cpp
        TestConnection.cpp

    Fixed up test factory to be less verbose. Cut down packet types.

    Fix TestNetworkInterface.cpp

    Separate DNS resolver implementation from resolver base class

    Move resolver tests into TestDNSResolver.cpp

    Separate BSD socket implementation from network interface base class.

    Keep 'test_network_interface' into TestNetworkInterface.cpp

    Move other tests into TestBSDSocketsInterface.cpp


Monday May 12th, 2014:
----------------------

    Implement the code that serializes the message types, and uses the 
    factory on the reading side to create the message by type, and then
    serializes the message.

    Does the message really need to include the id? Yep. It does.

    Was clobbering the message id to 0 inside serialize write.

    Seems to be something wrong with serialization or test harness.

    Message coming in with "sequence" (inside TestMessage) that is
    the last message sent, but it is in receive queue as 0 (and message id 0). 
    Almost like the message ptr is being reused, or flipped incorrectly at 
    some point. 

    What is going on?!

    Was just an inversion of the serialize for messageId. Was writing 0
    instead of the correct message id.

    Now the test harness works fine!


Sunday May 11th, 2014:
----------------------

    Added receive queue data and added receive queue to message channel

    There is now a receive message id starting at zero. Will try to receive
    this message id from receive queue, will block other messages until
    it is available.

    Added code to receive messages from queue.

    When a message channel data comes in add the messages in it to the receive queue.

    If any message in the packet data is newer than the current recieve message id,
    discard the packet before doing any work (otherwise, messages will be acked
    potentially, and never resent).

    If a message is older than the last receive message id, ignore it.

    Implement the counters so that the unit test can detect correct operation
    (eg. messages sent/received/read/written/discarded-too-new)

    Implement the code to serialize the message channel data on write, and receive.

    Need some way to pass the max messages per-packet, and other config data
    in to the message channel on serialize. How do to this? Allow the channel
    to pass it's data in via some opaque ptr in on serialize?

    Would be nice to have a *general* mechanism to do this, as you are going 
    to always want to access the config data while doing work for that channel.

    Settled on simply passing a pointer to the channel config in to the 
    channel data on creation. This makes the most sense in my opinion.


Monday May 5th, 2014:
---------------------

    Add code to gather n messages to include in packet serialize write

    Add code to ignore messages that have already been sent within "send rate" of current time.

    Don't send messages that are > send queue size / 2 ahead of oldest message in send queue.

    Ignore message packing. Assume that max # of messages in packet will not exceed max packet size.

    Added id to message. Makes a lot of other stuff easier vs. maintaining separate data structure
    to have the id and message ptr everywhere.

    Set the message id when it is queued for send.

    Implement the message channel data class.

    Add vector of messages to message channel data.

    Verify messages are correctly setup via prints.

    Need an "add fast" option for struct like entries for which copying is bad,

    eg. sent packet entry has a vector. I don't want to copy that!

    Added "InsertFast" method that just sets valid 1, sets sequence then returns entry pointer to fill.

    Add sliding window for packets "m_sentPackets" with a vector of size max messages per packet (uint16_t ids)

    Add code to process acks and walk the m_sentPackets, if not already acked, mark that packet acked, 
    and then walk over all messages in send queue and remove them from the send queue.

    Add logs to verify this is working properly on the send side.


Sunday May 4th, 2014:
---------------------

    Sketched out new "MessageChannel" class and supporting structures.

    Added new "Message" base with "BlockMessage" being hard-coded to type 0 (saves extra virtual for "IsBlock")

    Added new "TestMessage" with sequence #

    This should provide all I need to test the message channel is working.

    Added simple config for message channel.

    Add the message factory to the config.

    Sketched out test harness for message channel.

    Now implement "SendMessage", "ReceiveMessage" interface on channel and have failing test.

    Send queue needs to be adjusted to a sliding window.

    Added new function "HasSlotAvailable" to send queue to detect if we have overflowed it.


Saturday April 12th, 2014:
--------------------------

    Hook up the "resend rate", eg. don't send the block down the channel continuously
    resend it at some fixed rate, eg. 10 times per-second.

    This requires current id for block and a time since last send of that id.
    Set this block id initially to 0xFFFF and if it is different to current
    block, update block id and immediately send block, set time to 0.0

    Added simple network timebase. Used doubles to keep it simple.

    There seem to be some problems with ack processing for reliable blocks.

    Debug it with logs. Is the ack processing to blame, or the reliable block channel?

    Error appears to be in the ack system. It's off by one somewhere, eg. acking packet 10
    but packet 10 was not received (packet 10 was).

    Yes, there was an error in the generate ack bits function.


Thursday April 4th, 2014:
-------------------------

    Make sure an exception thrown while processing channel data discards the packet
    (ensures that packet does not get marked as acked).

    Added counter for packet process fail.

    Add a unit test to verify this is the case, eg. create a test channel that has
    this behavior, and verify that no packets are acked while sent through with
    the exception being thrown.


Sunday March 30th, 2014:
------------------------

    Sketched out new reliable channel, channel data classes.

    Reliable channel will initially only send only one block at a time.

    Initially just resend the most recent block until it is acked.

    Added sliding windor for sent packets. This provides constant time
    lookup from the acked sequence to the block id.

    Added code to pop the block from the send queue if the acked
    block id matches the front block id in the send queue.

    On the receive side, if the block is not the id we are expecting
    then the process packet should throw an exception when processing

    This exception should discard that packet so it does not get acked.

    If the recieve side has the block id it expects, then it should
    simply queue up that block and id into the receive queue.

    Implement unit test with simulated packet loss.

    Verify blocks get through reliably, and in-order, eg. block contents
    should be i, where i is the i'th packet sent.

    Extend unit test to verify generate ack bits is working correctly.

    Found a bug in sliding window. Was not clearing entries on reset properly.


Tuesday March 25th, 2014:
-------------------------

    Try to implement basic dumb channel.

    Serialize write is working. Everything up to including channel data
    in the packet, and then serializing this channel data with skip bits
    works fine.

    Added interface that lets connection packet:

        1. Get the number of channels
        2. Create data for a channel by channel #

    Made this a shared_ptr owned by the connection, that way it can be shared
    with the packets, even if the connection object itself is not a shared ptr.

    Serialize read is working, reading blocks.

    Next, process the data on serialize read.

    Verify send -> serialize write -> serialize read -> receive channel block.

    Clean up logs and adjust unit test to iterate until 10 blocks received.


Monday March 24th, 2014:
------------------------

    Added channel interface.

    Seeing as we don't actually serialize the data *at all* inside the connection
    we cannot directly serialize channel data from inside connection processing.

    Added new object "ChannelData" derives from "Object" so it has serialize.

    Then added new interface to channel to drive this data:

        virtual shared_ptr<ChannelData> CreateData() = 0;

        virtual shared_ptr<ChannelData> GetDataForPacket( uint16_t sequence ) = 0;

        virtual void ProcessDataFromPacket( shared_ptr<ChannelData> data ) = 0;

        virtual void ProcessAck( uint16_t ack ) = 0;

    This should provide enough information to implement any channel strategy.

        1. You can get a data object to include in a packet with sequence
        2. If there is no data available right now, return nullptr. (drives skip bits)
        3. On read, create data then serialize in -- once serialize has completed
           call process data.
        4. Process ack should provide enough to implement reliability

    For the channel to abort packet send or packet read, it should throw an exception.

    This exception will abort the entire packet read and discard the packet.


Sunday March 23rd, 2014:
------------------------

    Work out how the user adds channels and works with them?

    User creates the channels and adds them to the connection via shared ptr.

    User can then directly work with the channel specific interface,
    while the connection works with the base channel interface to 
    serialize read/write packets.

    This keeps the connection functionality separate from the channel
    specific functionality.

    I could have lots of different channel types, eg. JSON channel,
    CaptainProtoChannel, ProtocolBuffersChannel, RawChannel, EventChannel
    etc.

    The key is that the connection works with data blocks, and 
    aggregates them into a single packet with reliability.

    This is different to, and less efficient that past techniques
    I've done where the serialization is done entirely in-place.

    But it makes more sense and is easier to integrate different
    channel types into it, at the cost of only a small amount
    of inefficiency.

    Implement a simple channel serializer that just sends blocks of data
    eg. UnrelaibleBlockChannel -- stupid serialize sends bytes per-block, then
    block data. Only one block per-packet.

    This is a tracer bullet... what does the connection need from each
    channel in order to work properly?


Sunday March 16th, 2014:
------------------------

    Add sent packets to the sliding window

    Add received packets to the sliding window

    Extend sliding window to return false if packet is too old
    to add to sliding window, eg. discard packet

    Properly implement "sequence_less_than" and "sequence_greater_than" functions, 
    and test that they correctly handle 16 bit wrap around.

    Fixed bug in sliding window. Uninitialized data in ctor.

    Detect if received packet is too old to add to sliding window. Discard packet.

    Implement "GenerateAckBits" function.

    Unit test the generate ack bits function (it should basically take a sliding window and return ack, ack_bits)

    Yep. It was slightly off with or then shift. needed to be shift then or.

    Implement the "ProcessAcks" function to take ack + ack_bits and check for sent packets matching,
    call "PacketAcked" if it is the first time a sent packet with that sequence was acked.

    Add counters and include basics stats such as number of packets writter, read, acked, discarded.

    Extend unit test to verify counters match expected results, eg. acks especially.


Saturday March 15th, 2014:
--------------------------

    Should there be nodes? Yes. But strictly client/server. No support for P2P.

    This means there is just the server, with data per-client, and the client's
    view of the server data that is synchronized to it.

    So calling each connection a "client" instead of a node is probably correct.

    But a node is more generic, and it makes sense for a server to have n nodes,
    one per-client, but on the client to just have one node.

    Alternatively, it could be called "connection"?

    Yes. Going with connection. I can then later on create a server class
    with one connection per-client, and a client class with a single connection
    to the server. SORTED.

    Sketched out connection packet with sequence, ack, ack_bits.

    Sketched out functions and data structures required to implement reliability.

    Implement sliding window class.

    Maybe consider moving the sequence numbers, eg. most recent sent packet, most recent received *into* the sliding window.

    Maybe add a bool into sliding window, eg. have you ever had an entry inserted yet? (eg. for initial sliding window set sequence)

    Yes. Everything becomes simple if the sliding window class is well designed like this...

    Sliding window is basic. But it seems additional logic is required, like how to 
    detect if a received packet is in the window. Discarding of old packets etc.

    This logic I think is specific to the send and receive sliding windows.

    I will keep it out of the sliding window class.

    Unit test the sliding window class.


Friday March 14th, 2014:
------------------------

    Get the network interface working across IPv6 too ("::1")

    Now adjust so you can create the network interface in IPv4 mode or IPv6 mode.

    Currently using AF_INET to mean IPv4, AF_INET6 to mean IPv6 only and
    AF_UNSPEC to mean dual stack. This may not be the best.

    Do I care about dual stack at this point? No I don't. Cut dual stack support.

    Add query fn to get maximum packet type from factory. Necessary for serialization.


Wednesday March 12th, 2014:
---------------------------

    Add test to verify send and receive packet by address.

    Create non-blocking socket inside network interface.    

    Clean up socket in network interface dtor.

    Fixed address initialize from addrinfo to include port.

    Hack up the sendto to work with IPv6 address types.

    Added comparison operators to packets and learned about static_pointer_cast.

    Added code to verify the packets received on the other side verify the 
    packets that were sent, before serialize read and write.

    Hacked up a really nasty serialize write buffer for stream. I'll fix this up later.

    Implemented internal packet send code that calls through to sendto with
    serialize write buffer contents.

    Need a config to pass into the network interface, we'll need to specify
    the maximum packet size in there, otherwise we're going to not know
    what size buffer to use for recvfrom.

    Need to pass factory into network interface. It needs it so it knows
    what the maximum packet type is (for serialize).

    Added network interface config. config specifies max packet size,
    packet factory, dns resolver (optional) and UDP port to bind to.

    Call recvfrom and get packet data. 

    Some problem where recvfrom is always returning -1. Possible that my address is wrong
    and it is always WSAWOULDBLOCK? Need to dig in and get socket error in this case.

    Address int32 was double converted using htonl. Fixed.

    Serialize read this buffer into a stream and then determine the packet type.

    Create an object of the correct type. If packet for type cannot be created, 
    increment a counter and discard the packet.

    Serialize the packet. If the packet serialize fails discard the packet
    and increment serialize read failure counter.

    If the packet serializes properly, add the packet to the receive queue


Sunday March 9th, 2014:
-----------------------

    Implement lookup from hostname to address inside network interface via resolver

    Added counters to network interface to aid in unit tests (eg. num packets sent, received, discarded)

    Add test to verify hostname lookup + resolve send queue + send on resolve works

    Add test to verify hostname failure discards packets

    Carry the port # from send to address -> internal send queue (resolve). Verify with logs.


Tuesday March 5th, 2014:
------------------------

    Need a way to resolve DNS into addresses.

    Add a new interface "Resolver". 

    Derive from this and create a new "DNSResolver"

    This will keep it generic, for example I could create redis
    based server resolver for giving game instances server names,
    or a master server list etc.

    By having a resolver interface I won't care how the name -> address
    resolution occurs, just that it *does* occur.

    Added way to initialize address from addrinfo ptr (from resolve DNS...)

    Cache data on lookup so it can be queried.

    If an address has already been looked up, don't fire off a new async for it.
    (but must you call the callback?! not sure... probably.)

    There should have the option to clear the cache.

    There should be a way to look up an address without calling lambda.

    Optimize DNS resolver so that in-progress queries are in a separate hash,
    so we don't go O(n) in the udptae after lots of queries have been done.

    Verify that multiple overlapping resolve calls work, and all the lambdas are called on completion

    Verify resolve failure works as well, and cached fail result.


Sunday March 2nd, 2014:
-----------------------

    Basic serialization, serialize_int etc.

    Base object for serialization with virtual Serialize( Stream & stream ) method.

    Back packet class with serialize, address string (eg. sent from), and type int.

    Factory class to create packets from type.

    If you try to create an object of a type that is not registered, throw a runtime_error.

    Explore "getaddrinfo" for IPv6 aware hostname -> IP address etc.

    Implement basic address class capable of handling IPv6 and IPv6

    Implement function to convert IPv4 string to address int + port

    Verify IPv4 UDP broadcast address is converted properly: 0xffffffff

    Implement function to convert IPv4 address + port back to string
    (Omit the port if it is zero).

    Added code to initialize IPv6 with 16bit pairs, including htons
    (decided on network byte order for internal address data...)

    Handle port number in IPv6 output string, eg. "[v6addr]:portnum"

    Implement IPv6 parse from string

    Implement IPv6 parse from string with port, eg. "[addr6]:portnum"