Skip to content

Commit

Permalink
Release/r2018.06.15 (#191)
Browse files Browse the repository at this point in the history
* allow ptr in inline structs

* inline pointer_to_struce mode: update comments. return error on pointer not to struct

* fix(dbtest): Use os.Kill on windows instead of Interrupt 🐛

I've added a use for os.Kill, instead of os.Interrupt signal, when using
Windows. I'm current developing my project on Windows, and using
DBServer.Stop() was resulting in: "timeout waiting for mongod process to
die". After investigating, I've discovered that os.Interrupt isn't
implemented on Windows, and it seems golang has Frozen this issue due to
age (2013). They instruct to use os.Kill instead. Using this, the
DBServer on my project works with no problem.

* Respect nil slices, maps in bson encoder (#147)

* socket: only send client metadata once per socket (#105)

Periodic cluster synchronisation calls isMaster() which currently resends the
"client" metadata every call - the spec specifies:

	isMaster commands issued after the initial connection handshake MUST NOT
	contain handshake arguments

	https://github.com/mongodb/specifications/blob/master/source/mongodb-handshake/handshake.rst#connection-handshake

This hotfix prevents subsequent isMaster calls from sending the client metadata
again - fixes #101 and fixes #103.

Thanks to @changwoo-nam @qhenkart @canthefason @jyoon17 for spotting the initial
issue, opening tickets, and having the problem debugged with a PoC fix before I
even woke up.

* Merge Development (#111)

* Brings in a patch on having flusher not suppress errors. (#81)

go-mgo#360

* Fallback to JSON tags when BSON tag isn't present (#91)

* Fallback to JSON tags when BSON tag isn't present


Cleanup.

* Add test to demonstrate tagging fallback.

- Test coverage for tagging test.

* socket: only send client metadata once per socket

Periodic cluster synchronisation calls isMaster() which currently resends the
"client" metadata every call - the spec specifies:

	isMaster commands issued after the initial connection handshake MUST NOT
	contain handshake arguments

	https://github.com/mongodb/specifications/blob/master/source/mongodb-handshake/handshake.rst#connection-handshake

This hotfix prevents subsequent isMaster calls from sending the client metadata
again - fixes #101 and fixes #103.

Thanks to @changwoo-nam @qhenkart @canthefason @jyoon17 for spotting the initial
issue, opening tickets, and having the problem debugged with a PoC fix before I
even woke up.

* Cluster abended test 254 (#100)

* Add a test that mongo Server gets their abended reset as necessary.

See https://github.com/go-mgo/mgo/issues/254 and
https://github.com/go-mgo/mgo/pull/255/files

* Include the patch from Issue 255.

This brings in a test which fails without the patch, and passes with the
patch. Still to be tested, manual tcpkill of a socket.

* changeStream support (#97)

Add $changeStream support

* readme: credit @peterdeka and @steve-gray (#110)

* Hotfix #120  (#136)

* cluster: fix deadlock in cluster synchronisation (#120)

For a impressively thorough breakdown of the problem, see:
	#120 (comment)

Huge thanks to @dvic and @KJTsanaktsidis for the report and fix.

* readme: credit @dvic and @KJTsanaktsidis

* added support for marshalling/unmarshalling maps with non-string keys

* refactor method receiver

* added support for json-compatible support for slices and maps Marshal() func: nil slice or map converts to nil, not empty (initialized with len=0)

* fix IsNil on slices and maps

format

* added godoc

* fix sasl empty payload

* fix scram-sha-1 auth

* revert fix sasl empty payload

* Separate read/write network timeouts (#161)

* socket: separate read/write network timeouts

Splits DialInfo.Timeout (defaults to 60s when using mgo.Dial()) into ReadTimeout
and WriteTimeout to address #160. Read/write timeout defaults to
DialInfo.Timeout to preserve existing behaviour.

* cluster: remove AcquireSocket

Only used by tests, replaced by the pool-aware acquire socket functions:
	* AcquireSocketWithPoolTimeout
	* AcquireSocketWithBlocking

* cluster: use configured timeouts for cluster operations

* `mongoCluster.syncServer()` no longer uses hard-coded 5 seconds
* `mongoCluster.isMaster()` no longer uses hard-coded 10 seconds

* tests: use DialInfo for internal timeouts

* server: fix fantastic serverTags nil slice bug

When unmarshalling serverTags, it is now an empty slice, instead of a nil slice.

`len(thing) == 0` works all the time, regardless.

* cluster: remove unused duplicate pool config

* session: avoid calculating default values in hot path

Changes `DialWithInfo` to handle setting default values by setting the relevant
`DialInfo` field, rather than calling the respective methods in the hot path for:

	* `PoolLimit`
	* `ReadTimeout`
	* `WriteTimeout`

* session: remove unused consts

* session: update docs

* add URI options: "w", "j", "wtimeoutMS" (#162)

* add URI options: "w", "j", "wtimeoutMS"

* change "w" to "j"

* Add Collation support for calling Count() on a Query (#166)

* Expand documentation for *Iter.Next (#163)

The documentation now explains the difference between calling Err and
Close after Next returns false.

The example code has been expanded to include checking for timeout.

* add NewMongoTimestamp() and MongoTimestamp.Time(),Counter() (#171)

code is inspired by go-mgo#202

* MGO-156 Avoid iter.Next deadlock on dead sockets (#182)

* Allow passing slice pointer as an interface pointer to Iter.All (#181)

* socket: only send client metadata once per socket (#105)

Periodic cluster synchronisation calls isMaster() which currently resends the
"client" metadata every call - the spec specifies:

	isMaster commands issued after the initial connection handshake MUST NOT
	contain handshake arguments

	https://github.com/mongodb/specifications/blob/master/source/mongodb-handshake/handshake.rst#connection-handshake

This hotfix prevents subsequent isMaster calls from sending the client metadata
again - fixes #101 and fixes #103.

Thanks to @changwoo-nam @qhenkart @canthefason @jyoon17 for spotting the initial
issue, opening tickets, and having the problem debugged with a PoC fix before I
even woke up.

* Merge Development (#111)

* Brings in a patch on having flusher not suppress errors. (#81)

go-mgo#360

* Fallback to JSON tags when BSON tag isn't present (#91)

* Fallback to JSON tags when BSON tag isn't present


Cleanup.

* Add test to demonstrate tagging fallback.

- Test coverage for tagging test.

* socket: only send client metadata once per socket

Periodic cluster synchronisation calls isMaster() which currently resends the
"client" metadata every call - the spec specifies:

	isMaster commands issued after the initial connection handshake MUST NOT
	contain handshake arguments

	https://github.com/mongodb/specifications/blob/master/source/mongodb-handshake/handshake.rst#connection-handshake

This hotfix prevents subsequent isMaster calls from sending the client metadata
again - fixes #101 and fixes #103.

Thanks to @changwoo-nam @qhenkart @canthefason @jyoon17 for spotting the initial
issue, opening tickets, and having the problem debugged with a PoC fix before I
even woke up.

* Cluster abended test 254 (#100)

* Add a test that mongo Server gets their abended reset as necessary.

See https://github.com/go-mgo/mgo/issues/254 and
https://github.com/go-mgo/mgo/pull/255/files

* Include the patch from Issue 255.

This brings in a test which fails without the patch, and passes with the
patch. Still to be tested, manual tcpkill of a socket.

* changeStream support (#97)

Add $changeStream support

* readme: credit @peterdeka and @steve-gray (#110)

* Hotfix #120  (#136)

* cluster: fix deadlock in cluster synchronisation (#120)

For a impressively thorough breakdown of the problem, see:
	#120 (comment)

Huge thanks to @dvic and @KJTsanaktsidis for the report and fix.

* readme: credit @dvic and @KJTsanaktsidis

* Allow passing slice pointer as an interface pointer to Iter.All

* Reverted to original error message, added test case for interface{} ptr

* Contributing:findAndModify support writeConcern (#185)

* socket: only send client metadata once per socket (#105)

Periodic cluster synchronisation calls isMaster() which currently resends the
"client" metadata every call - the spec specifies:

	isMaster commands issued after the initial connection handshake MUST NOT
	contain handshake arguments

	https://github.com/mongodb/specifications/blob/master/source/mongodb-handshake/handshake.rst#connection-handshake

This hotfix prevents subsequent isMaster calls from sending the client metadata
again - fixes #101 and fixes #103.

Thanks to @changwoo-nam @qhenkart @canthefason @jyoon17 for spotting the initial
issue, opening tickets, and having the problem debugged with a PoC fix before I
even woke up.

* Merge Development (#111)

* Brings in a patch on having flusher not suppress errors. (#81)

go-mgo#360

* Fallback to JSON tags when BSON tag isn't present (#91)

* Fallback to JSON tags when BSON tag isn't present


Cleanup.

* Add test to demonstrate tagging fallback.

- Test coverage for tagging test.

* socket: only send client metadata once per socket

Periodic cluster synchronisation calls isMaster() which currently resends the
"client" metadata every call - the spec specifies:

	isMaster commands issued after the initial connection handshake MUST NOT
	contain handshake arguments

	https://github.com/mongodb/specifications/blob/master/source/mongodb-handshake/handshake.rst#connection-handshake

This hotfix prevents subsequent isMaster calls from sending the client metadata
again - fixes #101 and fixes #103.

Thanks to @changwoo-nam @qhenkart @canthefason @jyoon17 for spotting the initial
issue, opening tickets, and having the problem debugged with a PoC fix before I
even woke up.

* Cluster abended test 254 (#100)

* Add a test that mongo Server gets their abended reset as necessary.

See https://github.com/go-mgo/mgo/issues/254 and
https://github.com/go-mgo/mgo/pull/255/files

* Include the patch from Issue 255.

This brings in a test which fails without the patch, and passes with the
patch. Still to be tested, manual tcpkill of a socket.

* changeStream support (#97)

Add $changeStream support

* readme: credit @peterdeka and @steve-gray (#110)

* Hotfix #120  (#136)

* cluster: fix deadlock in cluster synchronisation (#120)

For a impressively thorough breakdown of the problem, see:
	#120 (comment)

Huge thanks to @dvic and @KJTsanaktsidis for the report and fix.

* readme: credit @dvic and @KJTsanaktsidis

* findAndModify support writeConcern

* fix

* readme: credit everyone (#187)

* @cedric-cordenier
* @DaytonG
* @ddspog
* @gedge
* @jefferickson
* @larrycinnabar
* @Mei-Zhao
* @roobre

* revert: MGO-156 Avoid iter.Next deadlock on dead sockets (#182) (#188)

This reverts commit 7253b2b.

* Add support for ssl dial string (#184)

* Add support for ssl dial string

* Ensure we dont override user settings

* update examples

* update ssl value parsing

* PingSsl test

* skip test requiring system certificates

* readme: credit @tbruyelle (#190)
  • Loading branch information
domodwyer committed Jun 15, 2018
1 parent efe0945 commit 113d396
Show file tree
Hide file tree
Showing 18 changed files with 818 additions and 218 deletions.
20 changes: 19 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ A [sub-package](https://godoc.org/github.com/globalsign/mgo/bson) that implement
* Supports dropping all indexes on a collection ([details](https://github.com/globalsign/mgo/pull/25))
* Annotates log entries/profiler output with optional appName on 3.4+ ([details](https://github.com/globalsign/mgo/pull/28))
* Support for read-only [views](https://docs.mongodb.com/manual/core/views/) in 3.4+ ([details](https://github.com/globalsign/mgo/pull/33))
* Support for [collations](https://docs.mongodb.com/manual/reference/collation/) in 3.4+ ([details](https://github.com/globalsign/mgo/pull/37))
* Support for [collations](https://docs.mongodb.com/manual/reference/collation/) in 3.4+ ([details](https://github.com/globalsign/mgo/pull/37), [more](https://github.com/globalsign/mgo/pull/166))
* Provide BSON constants for convenience/sanity ([details](https://github.com/globalsign/mgo/pull/41))
* Consistently unmarshal time.Time values as UTC ([details](https://github.com/globalsign/mgo/pull/42))
* Enforces best practise coding guidelines ([details](https://github.com/globalsign/mgo/pull/44))
Expand All @@ -49,6 +49,15 @@ A [sub-package](https://godoc.org/github.com/globalsign/mgo/bson) that implement
* Add BSON stream encoders ([details](https://github.com/globalsign/mgo/pull/127))
* Add integer map key support in the BSON encoder ([details](https://github.com/globalsign/mgo/pull/140))
* Support aggregation [collations](https://docs.mongodb.com/manual/reference/collation/) ([details](https://github.com/globalsign/mgo/pull/144))
* Support encoding of inline struct references ([details](https://github.com/globalsign/mgo/pull/146))
* Improved windows test harness ([details](https://github.com/globalsign/mgo/pull/158))
* Improved type and nil handling in the BSON codec ([details](https://github.com/globalsign/mgo/pull/147/files), [more](https://github.com/globalsign/mgo/pull/181))
* Separated network read/write timeouts ([details](https://github.com/globalsign/mgo/pull/161))
* Expanded dial string configuration options ([details](https://github.com/globalsign/mgo/pull/162))
* Implement MongoTimestamp ([details](https://github.com/globalsign/mgo/pull/171))
* Support setting `writeConcern` for `findAndModify` operations ([details](https://github.com/globalsign/mgo/pull/185))
* Add `ssl` to the dial string options ([details](https://github.com/globalsign/mgo/pull/184))


---

Expand All @@ -59,23 +68,32 @@ A [sub-package](https://godoc.org/github.com/globalsign/mgo/bson) that implement
* @BenLubar
* @carldunham
* @carter2000
* @cedric-cordenier
* @cezarsa
* @DaytonG
* @ddspog
* @drichelson
* @dvic
* @eaglerayp
* @feliixx
* @fmpwizard
* @gazoon
* @gedge
* @gnawux
* @idy
* @jameinel
* @jefferickson
* @johnlawsharrison
* @KJTsanaktsidis
* @larrycinnabar
* @mapete94
* @maxnoel
* @mcspring
* @Mei-Zhao
* @peterdeka
* @Reenjii
* @roobre
* @smoya
* @steve-gray
* @tbruyelle
* @wgallagher
41 changes: 40 additions & 1 deletion bson/bson.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ import (
"errors"
"fmt"
"io"
"math"
"os"
"reflect"
"runtime"
Expand Down Expand Up @@ -426,6 +427,36 @@ func Now() time.Time {
// strange reason has its own datatype defined in BSON.
type MongoTimestamp int64

// Time returns the time part of ts which is stored with second precision.
func (ts MongoTimestamp) Time() time.Time {
return time.Unix(int64(uint64(ts)>>32), 0)
}

// Counter returns the counter part of ts.
func (ts MongoTimestamp) Counter() uint32 {
return uint32(ts)
}

// NewMongoTimestamp creates a timestamp using the given
// date `t` (with second precision) and counter `c` (unique for `t`).
//
// Returns an error if time `t` is not between 1970-01-01T00:00:00Z
// and 2106-02-07T06:28:15Z (inclusive).
//
// Note that two MongoTimestamps should never have the same (time, counter) combination:
// the caller must ensure the counter `c` is increased if creating multiple MongoTimestamp
// values for the same time `t` (ignoring fractions of seconds).
func NewMongoTimestamp(t time.Time, c uint32) (MongoTimestamp, error) {
u := t.Unix()
if u < 0 || u > math.MaxUint32 {
return -1, errors.New("invalid value for time")
}

i := int64(u<<32 | int64(c))

return MongoTimestamp(i), nil
}

type orderKey int64

// MaxKey is a special value that compares higher than all other possible BSON
Expand Down Expand Up @@ -746,6 +777,14 @@ func getStructInfo(st reflect.Type) (*structInfo, error) {
return nil, errors.New("Option ,inline needs a map with string keys in struct " + st.String())
}
inlineMap = info.Num
case reflect.Ptr:
// allow only pointer to struct
if kind := field.Type.Elem().Kind(); kind != reflect.Struct {
return nil, errors.New("Option ,inline allows a pointer only to a struct, was given pointer to " + kind.String())
}

field.Type = field.Type.Elem()
fallthrough
case reflect.Struct:
sinfo, err := getStructInfo(field.Type)
if err != nil {
Expand All @@ -765,7 +804,7 @@ func getStructInfo(st reflect.Type) (*structInfo, error) {
fieldsList = append(fieldsList, finfo)
}
default:
panic("Option ,inline needs a struct value or map field")
panic("Option ,inline needs a struct value or a pointer to a struct or map field")
}
continue
}
Expand Down
167 changes: 158 additions & 9 deletions bson/bson_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,8 @@ import (
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"math/rand"
"net/url"
"reflect"
"strings"
Expand Down Expand Up @@ -271,6 +273,42 @@ func (s *S) TestMarshalBuffer(c *C) {
c.Assert(data, DeepEquals, buf[:len(data)])
}

func (s *S) TestPtrInline(c *C) {
cases := []struct {
In interface{}
Out bson.M
}{
{
In: inlinePtrStruct{A: 1, MStruct: &MStruct{M: 3}},
Out: bson.M{"a": 1, "m": 3},
},
{ // go deeper
In: inlinePtrPtrStruct{B: 10, inlinePtrStruct: &inlinePtrStruct{A: 20, MStruct: &MStruct{M: 30}}},
Out: bson.M{"b": 10, "a": 20, "m": 30},
},
{
// nil embed struct
In: &inlinePtrStruct{A: 3},
Out: bson.M{"a": 3},
},
{
// nil embed struct
In: &inlinePtrPtrStruct{B: 5},
Out: bson.M{"b": 5},
},
}

for _, cs := range cases {
data, err := bson.Marshal(cs.In)
c.Assert(err, IsNil)
var dataBSON bson.M
err = bson.Unmarshal(data, &dataBSON)
c.Assert(err, IsNil)

c.Assert(dataBSON, DeepEquals, cs.Out)
}
}

// --------------------------------------------------------------------------
// Some one way marshaling operations which would unmarshal differently.

Expand Down Expand Up @@ -713,8 +751,6 @@ var marshalErrorItems = []testItemType{
"Attempted to marshal empty Raw document"},
{bson.M{"w": bson.Raw{Kind: 0x3, Data: []byte{}}},
"Attempted to marshal empty Raw document"},
{&inlineCantPtr{&struct{ A, B int }{1, 2}},
"Option ,inline needs a struct value or map field"},
{&inlineDupName{1, struct{ A, B int }{2, 3}},
"Duplicated key 'a' in struct bson_test.inlineDupName"},
{&inlineDupMap{},
Expand Down Expand Up @@ -1171,8 +1207,19 @@ type inlineBadKeyMap struct {
M map[int]int `bson:",inline"`
}
type inlineUnexported struct {
M map[string]interface{} `bson:",inline"`
unexported `bson:",inline"`
M map[string]interface{} `bson:",inline"`
unexported `bson:",inline"`
}
type MStruct struct {
M int `bson:"m,omitempty"`
}
type inlinePtrStruct struct {
A int
*MStruct `bson:",inline"`
}
type inlinePtrPtrStruct struct {
B int
*inlinePtrStruct `bson:",inline"`
}
type unexported struct {
A int
Expand Down Expand Up @@ -1229,11 +1276,11 @@ func (s ifaceSlice) GetBSON() (interface{}, error) {

type (
MyString string
MyBytes []byte
MyBool bool
MyD []bson.DocElem
MyRawD []bson.RawDocElem
MyM map[string]interface{}
MyBytes []byte
MyBool bool
MyD []bson.DocElem
MyRawD []bson.RawDocElem
MyM map[string]interface{}
)

var (
Expand Down Expand Up @@ -1888,3 +1935,105 @@ func (s *S) BenchmarkNewObjectId(c *C) {
bson.NewObjectId()
}
}

func (s *S) TestMarshalRespectNil(c *C) {
type T struct {
Slice []int
SlicePtr *[]int
Ptr *int
Map map[string]interface{}
MapPtr *map[string]interface{}
}

bson.SetRespectNilValues(true)
defer bson.SetRespectNilValues(false)

testStruct1 := T{}

c.Assert(testStruct1.Slice, IsNil)
c.Assert(testStruct1.SlicePtr, IsNil)
c.Assert(testStruct1.Map, IsNil)
c.Assert(testStruct1.MapPtr, IsNil)
c.Assert(testStruct1.Ptr, IsNil)

b, _ := bson.Marshal(testStruct1)

testStruct2 := T{}

bson.Unmarshal(b, &testStruct2)

c.Assert(testStruct2.Slice, IsNil)
c.Assert(testStruct2.SlicePtr, IsNil)
c.Assert(testStruct2.Map, IsNil)
c.Assert(testStruct2.MapPtr, IsNil)
c.Assert(testStruct2.Ptr, IsNil)

testStruct1 = T{
Slice: []int{},
SlicePtr: &[]int{},
Map: map[string]interface{}{},
MapPtr: &map[string]interface{}{},
}

c.Assert(testStruct1.Slice, NotNil)
c.Assert(testStruct1.SlicePtr, NotNil)
c.Assert(testStruct1.Map, NotNil)
c.Assert(testStruct1.MapPtr, NotNil)

b, _ = bson.Marshal(testStruct1)

testStruct2 = T{}

bson.Unmarshal(b, &testStruct2)

c.Assert(testStruct2.Slice, NotNil)
c.Assert(testStruct2.SlicePtr, NotNil)
c.Assert(testStruct2.Map, NotNil)
c.Assert(testStruct2.MapPtr, NotNil)
}

func (s *S) TestMongoTimestampTime(c *C) {
t := time.Now()
ts, err := bson.NewMongoTimestamp(t, 123)
c.Assert(err, IsNil)
c.Assert(ts.Time().Unix(), Equals, t.Unix())
}

func (s *S) TestMongoTimestampCounter(c *C) {
rnd := rand.Uint32()
ts, err := bson.NewMongoTimestamp(time.Now(), rnd)
c.Assert(err, IsNil)
c.Assert(ts.Counter(), Equals, rnd)
}

func (s *S) TestMongoTimestampError(c *C) {
t := time.Date(1969, time.December, 31, 23, 59, 59, 999, time.UTC)
ts, err := bson.NewMongoTimestamp(t, 321)
c.Assert(int64(ts), Equals, int64(-1))
c.Assert(err, ErrorMatches, "invalid value for time")
}

func ExampleNewMongoTimestamp() {

var counter uint32 = 1
var t time.Time

for i := 1; i <= 3; i++ {

if c := time.Now(); t.Unix() == c.Unix() {
counter++
} else {
t = c
counter = 1
}

ts, err := bson.NewMongoTimestamp(t, counter)
if err != nil {
fmt.Printf("NewMongoTimestamp error: %v", err)
} else {
fmt.Printf("NewMongoTimestamp encoded timestamp: %d\n", ts)
}

time.Sleep(500 * time.Millisecond)
}
}
15 changes: 14 additions & 1 deletion bson/compatibility.go
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
package bson

// Current state of the JSON tag fallback option.
// Current state of the JSON tag fallback option.
var useJSONTagFallback = false
var useRespectNilValues = false

// SetJSONTagFallback enables or disables the JSON-tag fallback for structure tagging. When this is enabled, structures
// without BSON tags on a field will fall-back to using the JSON tag (if present).
Expand All @@ -14,3 +15,15 @@ func SetJSONTagFallback(state bool) {
func JSONTagFallbackState() bool {
return useJSONTagFallback
}

// SetRespectNilValues enables or disables serializing nil slices or maps to `null` values.
// In other words it enables `encoding/json` compatible behaviour.
func SetRespectNilValues(state bool) {
useRespectNilValues = state
}

// RespectNilValuesState returns the current status of the JSON nil slices and maps fallback compatibility option.
// See SetRespectNilValues for more information.
func RespectNilValuesState() bool {
return useRespectNilValues
}
Loading

0 comments on commit 113d396

Please sign in to comment.