v0.0.5

Improved binary slicing and bit reading, avro_ocf decoder (thanks @xentripetal), asn1_ber decoder, renamed display aliases, new grep_by and paste function.

Changes

• Big internal bit reader refactor. Now much more consistent code and fixes some issues reading and decoding of binary arrays and binary slices. #123
  • Bit reading and IO have been moved to a bitio package.
  • Non-simple bit reading have been move out of bitio to decode package.
  • [0,1,1,0,0,1,1,0,0,1,1,1,0,0,0,1 | tobits] | tobytes | tostring returns "fq".
  • [.frames[0], .frames[-1]] | mp3 decode mp3 based on first and last frame of other mp3.
• Add grep_by that recursively selects using a filter condition and ignores errors. #102
  • grep_by(.type == "trak") finds all objects where .type is "trak" (all mp4 track boxes).
  • grep_by(tonumber | . >= 40 and . <= 100) find all numbers between 40 and 100.
  • grep_by(format == "jpeg") find all jpegs.
• Add paste function to read string from stdin util ^D (EOF). Can be used to paste in REPl etc. #143
  • paste | frompem | asn1_ber | repl wait for PEM encoded text on stdin (paste and press ^D), decode it with asn1_ber and start a sub-REPL with the result.
  • paste | fromjson decode pasted JSON.
  • eval(paste) eval pasted jq expression.
• Cleanup display aliases. Remove v and f, add da, dd, dv and ddv. #112
  • d/d($opts) display value and truncate long arrays and buffers
  • da/da($opts) display value and don't truncate arrays
  • dd/dd($opts) display value and don't truncate arrays or buffers
  • dv/dv($opts) verbosely display value and don't truncate arrays but truncate buffers
  • ddv/ddv($opts) verbosely display value and don't truncate arrays or buffers
• Refactor radix into toradix($base)/fromradix($base). #139
• Remove number_to_bytes. Can be done with tobytes. #139
• Change tobytes to zero pad most significant bits to byte alignment if needed. #133
• Add tobytes/tobits variant that takes an argument to add extra padding. #133
  • 0xf | tobytes 8 bit binary with last 4 bits set
  • 0xf | tobytes(4) 32 bit binary with last 4 bits set
  • 0xf | tobits(12) 12 bit binary with last 4 bits set
• Rename fq type buffer to binary as it makes more sense. #133
• Add topem/frompem to work with PEM encoding. #92
• Add Windows scoop install. #137 Thanks @thushan
• Add display, decode value, binary and binary array documentation. #118 #136 #133
• Add decode API documentation. #149
• Improved REPL completion for keys. #144
• Add -o force=<bool> option that sets force decode option. Same as mp4({force: true}). #126

Decoder changes

• avro_ocf Add decoder. #38 Thanks @xentripetal
  • Full avro OCF support. Handles all primitive, complex, and logical types besides decimals.
  • Able to handle deflate, snappy, and null codecs for blocks.
• asn1_ber Add decoder. #92
  • Also decodes CER and DER (X.690) but with no additional validation at the moment.
  • Support all types but real type is currently limited to range for 64 bit interger/float.
  • Has torepr support.
  • No schema support.
• aac_frame Only decode object types we know about. #130
• mp3 Shorter sync find heuristics. #104
• mp4 Add stz2 support
• mp4 Add pnot (preview container) and jP (JPEG 2000) signature. #125

Also thanks to @Doctor-love for keeping things tidy.

Changelog

• 6fc1efd Add test case with all data types
• ae4a624 Adds Windows Scoop instructions for fq.
• 4b809a7 Change avro codec to funcs
• 66ca1f1 Change tests to use new verbose syntax
• 7345b8c Cleanup
• 07ddf36 Cleanup for linting
• 4508241 Cleanup snappy
• 0909fb6 Comment on snappy decompression
• 21cfc70 Dates need to specify UTC too
• 75b8496 Fix lint
• 7a8e3ca Hook into registry, add codecs
• 251053e Initial pass on logical types
• 2605bce Lint and add basic doc
• ee18407 Parse header using avro decoders. Still not certain this is the best idea. Will get opinions before finalizing.
• ab50088 Polish of problem template and clarifying questions
• 27789f2 Regenerate docs
• 5a1d35e Remove redudant question and fix typo
• 31c4c0d Support snappy and deflate codecs
• 0300c95 Take heading off doc to match make doc format
• 6f57cdb Timestamps should be UTC
• 06085a2 Undo change to doc/file.mp4. I have no idea how this got changed in the first place? Maybe some macos shenanigans.
• d137a72 Update docker-golang from 1.17.6 to 1.17.7
• 267e30e Update github-go-version from 1.17.6, 1.17.6 to 1.17.7
• 1e859cd Update github-golangci-lint from 1.43.0 to 1.44.0
• 16849c8 Update linting
• d02d896 Update make-golangci-lint from 1.43.0 to 1.44.0
• 68e85a2 Use existing scalar description helper
• 3bab3d6 aac_frame: Only try decode object types we know about
• 0829c16 asn1_ber: Add decoder
• 0312c92 asn1_ber: Add more doc and multiple outputs for frompem
• 06245d1 binary,decode,doc: Rename buffer to binary and add some documentation
• 7c52153 bitio,decode: Refactor bitio usage and make buffer slicing more correct
• 0d74e87 bitio,doc: Even more cleanup
• d854ed5 bitio: Cleanup documentation a bit
• 82aeb35 bitio: More doc cleanup
• 01ecde6 bump: Add snappy config
• de64a99 cleanup some docs, change enum to mapper, error zigzag on more than 8 bytes
• 6cd1c38 decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers
• 4ab6381 decode: Add scalars args to FieldRootBitBuf
• be71eb0 decode: Rename LenFn() to LimitedFn, add FramedFn and document
• 7bc2521 doc,interp: Add some example usages to cli help
• 8e47fb1 doc,matroska: Fix filesname in example
• c15f528 doc: Add format links to format table
• b86da7a doc: Add inital decoder API documentation
• 49c90f8 doc: Add macOS security notes and move supported format up a bit
• 06b67e4 doc: Add more license details
• a8664ed doc: Add per format documentation
• 0955262 doc: Add snappy license
• 3630785 doc: Cleanup and add more decode value and binary documentation
• 710c29b doc: Color edges in diagram based on dest
• f0ce717 doc: Document display and some more jq hints
• b350468 doc: More decode API details and polish
• 6b51b06 doc: More display alias leftover fixes
• dd3e40f doc: Unbreak formats_digaram.jq since radix change
• c52a1a2 doc: Use f($a; $b) instead of jq f/2 notation
• 233d86a fq: Add arch and os to --version
• b8efd8e fuzz: Fuzz all formats
• e1bdfdf fuzz: List seed numbers and make it build again
• 6090b65 fuzz: Make it compile again and run one format per fuzz
• aea4884 github: Add basic issue template
• b55ca2c gojq: Rebase fq branch
• 47c978e goreleaser: Use zip for macos
• 8537117 id3v2: Should assert not validate magic
• d6ca481 initial work for avro OCF files
• ca68e6a interp: Add Platform() method to OS interface
• e792598 interp: Add grep_by/1 to recursively match using a filter
• 0a1a561 interp: Add missing default opts for tovalue
• 48a19cb interp: Add paste function to allow pasting text into REPL etc
• fc0aacb interp: Cleanup display aliases, now: d, da, dd, dv, ddv
• bf7a483 interp: Fix handling of group decode error from stdin
• 26d9650 interp: Refactor radix* into toradix($base)/fromradix($base)
• 366f6b1 interp: Support force decode as -o force=true
• 77ab667 interp: Use absolute path in errors
• c31ec2a interp: Use correct sym color
• 898dfec lint: Fix typeassert and case exhaustive warnings
• d540116 make doc
• bf170be make: Cleanup some not very used targets
• 8d2d88f mp3: Decrease max sync seek length between frames to 4k
• d555c32 mp4,fuzz: Fatal error on infinite sgpd box entries
• 45b00aa mp4: Add stz2 support
• 092609b mp4: Add video preview (pnot) and JPEG 2000 (jP) signatures
• febce5a mpeg_spu: Fatal error on infinite loop
• c58ba28 mpeg_spu: Fatal error on unknown cmd
• d1943da pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
• 2ab395a protobuf: Add note about sub message decoding
• af05381 repl,interp: Make stdio work during completion
• bd9be2c repl: Fix completion of non-underscore extkeys
• 69c745d simplify scalar usage
• 778a1a4 zip: Assert signature not validate

(Some commits have been removed from list for clarity)