mirror of
https://github.com/ergochat/ergo.git
synced 2024-12-23 03:02:48 +01:00
upgrade buntdb
This commit is contained in:
parent
1389d89a9b
commit
c5a9916302
2
go.mod
2
go.mod
@ -17,7 +17,7 @@ require (
|
||||
github.com/onsi/ginkgo v1.12.0 // indirect
|
||||
github.com/onsi/gomega v1.9.0 // indirect
|
||||
github.com/stretchr/testify v1.4.0 // indirect
|
||||
github.com/tidwall/buntdb v1.2.3
|
||||
github.com/tidwall/buntdb v1.2.6
|
||||
github.com/toorop/go-dkim v0.0.0-20201103131630-e1cd1a0a5208
|
||||
github.com/xdg-go/scram v1.0.2
|
||||
golang.org/x/crypto v0.0.0-20210415154028-4f45737414dc
|
||||
|
8
go.sum
8
go.sum
@ -41,12 +41,20 @@ github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJy
|
||||
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
|
||||
github.com/tidwall/btree v0.4.2 h1:aLwwJlG+InuFzdAPuBf9YCAR1LvSQ9zhC5aorFPlIPs=
|
||||
github.com/tidwall/btree v0.4.2/go.mod h1:huei1BkDWJ3/sLXmO+bsCNELL+Bp2Kks9OLyQFkzvA8=
|
||||
github.com/tidwall/btree v0.6.0 h1:JLYAFGV+1gjyFi3iQbO/fupBin+Ooh7dxqVV0twJ1Bo=
|
||||
github.com/tidwall/btree v0.6.0/go.mod h1:TzIRzen6yHbibdSfK6t8QimqbUnoxUSrZfeW7Uob0q4=
|
||||
github.com/tidwall/buntdb v1.2.3 h1:AoGVe4yrhKmnEPHrPrW5EUOATHOCIk4VtFvd8xn/ZtU=
|
||||
github.com/tidwall/buntdb v1.2.3/go.mod h1:+i/gBwYOHWG19wLgwMXFLkl00twh9+VWkkaOhuNQ4PA=
|
||||
github.com/tidwall/buntdb v1.2.6 h1:eS0QSmzHfCKjxxYGh8eH6wnK5VLsJ7UjyyIr29JmnEg=
|
||||
github.com/tidwall/buntdb v1.2.6/go.mod h1:zpXqlA5D2772I4cTqV3ifr2AZihDgi8FV7xAQu6edfc=
|
||||
github.com/tidwall/gjson v1.7.4 h1:19cchw8FOxkG5mdLRkGf9jqIqEyqdZhPqW60XfyFxk8=
|
||||
github.com/tidwall/gjson v1.7.4/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk=
|
||||
github.com/tidwall/gjson v1.8.0 h1:Qt+orfosKn0rbNTZqHYDqBrmm3UDA4KRkv70fDzG+PQ=
|
||||
github.com/tidwall/gjson v1.8.0/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk=
|
||||
github.com/tidwall/grect v0.1.1 h1:+kMEkxhoqB7rniVXzMEIA66XwU07STgINqxh+qVIndY=
|
||||
github.com/tidwall/grect v0.1.1/go.mod h1:CzvbGiFbWUwiJ1JohXLb28McpyBsI00TK9Y6pDWLGRQ=
|
||||
github.com/tidwall/grect v0.1.2 h1:wKVeQVZhjaFCKTTlpkDe3Ex4ko3cMGW3MRKawRe8uQ4=
|
||||
github.com/tidwall/grect v0.1.2/go.mod h1:v+n4ewstPGduVJebcp5Eh2WXBJBumNzyhK8GZt4gHNw=
|
||||
github.com/tidwall/lotsa v1.0.2 h1:dNVBH5MErdaQ/xd9s769R31/n2dXavsQ0Yf4TMEHHw8=
|
||||
github.com/tidwall/lotsa v1.0.2/go.mod h1:X6NiU+4yHA3fE3Puvpnn1XMDrFZrE9JO2/w+UMuqgR8=
|
||||
github.com/tidwall/match v1.0.3 h1:FQUVvBImDutD8wJLN6c5eMzWtjgONK9MwIBCOrUJKeE=
|
||||
|
43
vendor/github.com/tidwall/btree/PATH_HINT.md
generated
vendored
Normal file
43
vendor/github.com/tidwall/btree/PATH_HINT.md
generated
vendored
Normal file
@ -0,0 +1,43 @@
|
||||
# B-tree Path Hints
|
||||
|
||||
I use a thing I call path hints in my B-tree [C](https://github.com/tidwall/btree.c) and [Go](https://github.com/tidwall/btree) implementations. It's a search optimization.
|
||||
|
||||
## The B-tree
|
||||
|
||||
A standard [B-tree](https://en.wikipedia.org/wiki/B-tree) is an ordered tree-based data structure that stores its items in nodes. The B-tree has a single root node, which may have children nodes, and those children nodes may also have children nodes.
|
||||
|
||||
<img width="322" alt="image" src="https://user-images.githubusercontent.com/1156077/127664015-14ca38bb-1a3b-4d2f-80ff-27be0bd3d886.png">
|
||||
|
||||
Searching for items in a B-tree is fast. [O(log N)](https://en.wikipedia.org/wiki/Big_O_notation) to be exact.
|
||||
This is because the [binary search algorithm](https://en.wikipedia.org/wiki/Binary_search_algorithm) is used.
|
||||
|
||||
A binary search works by first comparing the item at the middle-most index of the root node with the target item.
|
||||
If the middle item is greater than the target item, then it divides the node in two and does the binary search on the left part of the node. If the middle is less, it searches the right part. And so on. If the target item is found, then the search stop. If the item is not found, then the search is passed to the child node at the appropriate index. This traversal terminates when item is found or there are no more child nodes.
|
||||
|
||||
<img width="600" alt="image" src="https://user-images.githubusercontent.com/1156077/127664822-6ab4f8f6-8ab5-477e-8e17-f52346f02819.png">
|
||||
|
||||
## The Path
|
||||
|
||||
Each index is a component of the path to the item (or where the item should be stored, if it does not exist in the tree).
|
||||
|
||||
Take the first example image. The item 9 is at path “1/0”. The item 16 is at path “1”. The item 21 is at path “2/1”. The item 5 is at path “0/2”.
|
||||
|
||||
## The Path Hint
|
||||
|
||||
A Path Hint is a predefined path that is provided to B-tree operations. It’s just a hint that says, “Hey B-tree, instead of starting your binary search with the middle index, start with what I provide you. My path may be wrong, and if so please provide me with the correct path so I get it right the next time.”
|
||||
|
||||
I’ve found using path hints can lead to a little performance increase of 150% - 300%. This is because in real-world cases the items that I’m working with are usually nearby each other in the tree.
|
||||
|
||||
Take for example inserting a group of timeseries points. They may often be received as chucks of near-contiguous items.
|
||||
Or, I'm sequentially inserting an ordered group of rows somewhere in the middle of a table.
|
||||
Or, I have a Redis-style key/value store, where the keys look have the common structure “user:98512:name”, “user:98512:email”, and I want to update a bunch of values for specified user.
|
||||
Using a path hint may help to avoid the unnecessary binary searching in each of these examples.
|
||||
|
||||
While I may see a 3x boost in when the path hint is right on, I'll only see around 5% decrease when the path hint is totally wrong.
|
||||
|
||||
## Using a Path Hint
|
||||
|
||||
For single-threaded programs, it’s possible to use one shared path hint per B-tree for the life of the program.
|
||||
For multi-threaded programs, I find it best to use one path hint per B-tree , per thread.
|
||||
For server-client programs, one path hint per B-tree, per client should suffice.
|
||||
|
38
vendor/github.com/tidwall/btree/README.md
generated
vendored
38
vendor/github.com/tidwall/btree/README.md
generated
vendored
@ -9,7 +9,7 @@ An [efficient](#performance) [B-tree](https://en.wikipedia.org/wiki/B-tree) impl
|
||||
- `Copy()` method with copy-on-write support.
|
||||
- Fast bulk loading for pre-ordered data using the `Load()` method.
|
||||
- All operations are thread-safe.
|
||||
- Path hinting optimization for operations with nearby keys.
|
||||
- [Path hinting](PATH_HINT.md) optimization for operations with nearby keys.
|
||||
|
||||
## Installing
|
||||
|
||||
@ -155,7 +155,7 @@ DeleteHint(item, *hint) # delete an item
|
||||
|
||||
This implementation was designed with performance in mind.
|
||||
|
||||
The following benchmarks were run on my 2019 Macbook Pro (2.4 GHz 8-Core Intel Core i9) using Go 1.15.3. The items are simple 8-byte ints.
|
||||
The following benchmarks were run on my 2019 Macbook Pro (2.4 GHz 8-Core Intel Core i9) using Go 1.16.5. The items are simple 8-byte ints.
|
||||
|
||||
- `google`: The [google/btree](https://github.com/google/btree) package
|
||||
- `tidwall`: The [tidwall/btree](https://github.com/tidwall/btree) package
|
||||
@ -163,29 +163,29 @@ The following benchmarks were run on my 2019 Macbook Pro (2.4 GHz 8-Core Intel C
|
||||
|
||||
```
|
||||
** sequential set **
|
||||
google: set-seq 1,000,000 ops in 160ms, 6,262,097/sec, 159 ns/op, 31.0 MB, 32 bytes/op
|
||||
tidwall: set-seq 1,000,000 ops in 142ms, 7,020,721/sec, 142 ns/op, 36.6 MB, 38 bytes/op
|
||||
tidwall: set-seq-hint 1,000,000 ops in 87ms, 11,503,315/sec, 86 ns/op, 36.6 MB, 38 bytes/op
|
||||
tidwall: load-seq 1,000,000 ops in 37ms, 27,177,242/sec, 36 ns/op, 36.6 MB, 38 bytes/op
|
||||
go-arr: append 1,000,000 ops in 49ms, 20,574,760/sec, 48 ns/op
|
||||
google: set-seq 1,000,000 ops in 163ms, 6,140,597/sec, 162 ns/op, 30.9 MB, 32 bytes/op
|
||||
tidwall: set-seq 1,000,000 ops in 141ms, 7,075,240/sec, 141 ns/op, 36.6 MB, 38 bytes/op
|
||||
tidwall: set-seq-hint 1,000,000 ops in 79ms, 12,673,902/sec, 78 ns/op, 36.6 MB, 38 bytes/op
|
||||
tidwall: load-seq 1,000,000 ops in 40ms, 24,887,293/sec, 40 ns/op, 36.6 MB, 38 bytes/op
|
||||
go-arr: append 1,000,000 ops in 51ms, 19,617,269/sec, 50 ns/op
|
||||
|
||||
** random set **
|
||||
google: set-rand 1,000,000 ops in 606ms, 1,649,921/sec, 606 ns/op, 21.5 MB, 22 bytes/op
|
||||
tidwall: set-rand 1,000,000 ops in 543ms, 1,841,590/sec, 543 ns/op, 26.7 MB, 27 bytes/op
|
||||
tidwall: set-rand-hint 1,000,000 ops in 573ms, 1,745,624/sec, 572 ns/op, 26.4 MB, 27 bytes/op
|
||||
tidwall: set-again 1,000,000 ops in 452ms, 2,212,581/sec, 451 ns/op, 27.1 MB, 28 bytes/op
|
||||
tidwall: set-after-copy 1,000,000 ops in 472ms, 2,117,457/sec, 472 ns/op, 27.9 MB, 29 bytes/op
|
||||
tidwall: load-rand 1,000,000 ops in 551ms, 1,816,498/sec, 550 ns/op, 26.1 MB, 27 bytes/op
|
||||
google: set-rand 1,000,000 ops in 666ms, 1,501,583/sec, 665 ns/op, 21.5 MB, 22 bytes/op
|
||||
tidwall: set-rand 1,000,000 ops in 569ms, 1,756,845/sec, 569 ns/op, 26.7 MB, 27 bytes/op
|
||||
tidwall: set-rand-hint 1,000,000 ops in 670ms, 1,491,637/sec, 670 ns/op, 26.4 MB, 27 bytes/op
|
||||
tidwall: set-again 1,000,000 ops in 488ms, 2,050,667/sec, 487 ns/op, 27.1 MB, 28 bytes/op
|
||||
tidwall: set-after-copy 1,000,000 ops in 494ms, 2,022,980/sec, 494 ns/op, 27.9 MB, 29 bytes/op
|
||||
tidwall: load-rand 1,000,000 ops in 594ms, 1,682,937/sec, 594 ns/op, 26.1 MB, 27 bytes/op
|
||||
|
||||
** sequential get **
|
||||
google: get-seq 1,000,000 ops in 133ms, 7,497,604/sec, 133 ns/op
|
||||
tidwall: get-seq 1,000,000 ops in 110ms, 9,082,972/sec, 110 ns/op
|
||||
tidwall: get-seq-hint 1,000,000 ops in 55ms, 18,289,945/sec, 54 ns/op
|
||||
google: get-seq 1,000,000 ops in 141ms, 7,078,690/sec, 141 ns/op
|
||||
tidwall: get-seq 1,000,000 ops in 124ms, 8,075,925/sec, 123 ns/op
|
||||
tidwall: get-seq-hint 1,000,000 ops in 40ms, 25,142,979/sec, 39 ns/op
|
||||
|
||||
** random get **
|
||||
google: get-rand 1,000,000 ops in 149ms, 6,704,337/sec, 149 ns/op
|
||||
tidwall: get-rand 1,000,000 ops in 131ms, 7,616,296/sec, 131 ns/op
|
||||
tidwall: get-rand-hint 1,000,000 ops in 216ms, 4,632,532/sec, 215 ns/op
|
||||
google: get-rand 1,000,000 ops in 152ms, 6,593,518/sec, 151 ns/op
|
||||
tidwall: get-rand 1,000,000 ops in 128ms, 7,783,293/sec, 128 ns/op
|
||||
tidwall: get-rand-hint 1,000,000 ops in 135ms, 7,403,823/sec, 135 ns/op
|
||||
```
|
||||
|
||||
*You can find the benchmark utility at [tidwall/btree-benchmark](https://github.com/tidwall/btree-benchmark)*
|
||||
|
392
vendor/github.com/tidwall/btree/btree.go
generated
vendored
392
vendor/github.com/tidwall/btree/btree.go
generated
vendored
@ -4,9 +4,11 @@
|
||||
|
||||
package btree
|
||||
|
||||
import "sync"
|
||||
import (
|
||||
"sync"
|
||||
)
|
||||
|
||||
const maxItems = 255
|
||||
const maxItems = 255 // max items per node. max children is +1
|
||||
const minItems = maxItems * 40 / 100
|
||||
|
||||
type cow struct {
|
||||
@ -17,18 +19,19 @@ type node struct {
|
||||
cow *cow
|
||||
leaf bool
|
||||
numItems int16
|
||||
count int
|
||||
items [maxItems]interface{}
|
||||
children *[maxItems + 1]*node
|
||||
}
|
||||
|
||||
// BTree is an ordered set items
|
||||
type BTree struct {
|
||||
mu *sync.RWMutex
|
||||
cow *cow
|
||||
root *node
|
||||
length int
|
||||
less func(a, b interface{}) bool
|
||||
lnode *node
|
||||
mu *sync.RWMutex
|
||||
cow *cow
|
||||
root *node
|
||||
count int
|
||||
less func(a, b interface{}) bool
|
||||
locks bool
|
||||
}
|
||||
|
||||
func (tr *BTree) newNode(leaf bool) *node {
|
||||
@ -43,17 +46,32 @@ func (tr *BTree) newNode(leaf bool) *node {
|
||||
// PathHint is a utility type used with the *Hint() functions. Hints provide
|
||||
// faster operations for clustered keys.
|
||||
type PathHint struct {
|
||||
used [8]bool
|
||||
path [8]uint8
|
||||
}
|
||||
|
||||
// New returns a new BTree
|
||||
func New(less func(a, b interface{}) bool) *BTree {
|
||||
return newBTree(less, true)
|
||||
}
|
||||
|
||||
// NewNonConcurrent returns a new BTree which is not safe for concurrent
|
||||
// write operations by multiple goroutines.
|
||||
//
|
||||
// This is useful for when you do not need the BTree to manage the locking,
|
||||
// but would rather do it yourself.
|
||||
func NewNonConcurrent(less func(a, b interface{}) bool) *BTree {
|
||||
return newBTree(less, false)
|
||||
}
|
||||
|
||||
func newBTree(less func(a, b interface{}) bool, locks bool) *BTree {
|
||||
if less == nil {
|
||||
panic("nil less")
|
||||
}
|
||||
tr := new(BTree)
|
||||
tr.mu = new(sync.RWMutex)
|
||||
tr.less = less
|
||||
tr.locks = locks
|
||||
return tr
|
||||
}
|
||||
|
||||
@ -68,18 +86,32 @@ func (n *node) find(key interface{}, less func(a, b interface{}) bool,
|
||||
) (index int16, found bool) {
|
||||
low := int16(0)
|
||||
high := n.numItems - 1
|
||||
if hint != nil && depth < 8 {
|
||||
if hint != nil && depth < 8 && hint.used[depth] {
|
||||
index = int16(hint.path[depth])
|
||||
if index > n.numItems-1 {
|
||||
if index >= n.numItems {
|
||||
// tail item
|
||||
if less(n.items[n.numItems-1], key) {
|
||||
if less(key, n.items[n.numItems-1]) {
|
||||
index = n.numItems - 1
|
||||
found = true
|
||||
goto path_match
|
||||
} else {
|
||||
index = n.numItems
|
||||
goto path_match
|
||||
}
|
||||
}
|
||||
index = n.numItems - 1
|
||||
}
|
||||
if less(key, n.items[index]) {
|
||||
if index == 0 || less(n.items[index-1], key) {
|
||||
goto path_match
|
||||
}
|
||||
high = index - 1
|
||||
} else if less(n.items[index], key) {
|
||||
low = index + 1
|
||||
} else {
|
||||
found = true
|
||||
goto done
|
||||
goto path_match
|
||||
}
|
||||
}
|
||||
for low <= high {
|
||||
@ -97,13 +129,16 @@ func (n *node) find(key interface{}, less func(a, b interface{}) bool,
|
||||
index = low
|
||||
found = false
|
||||
}
|
||||
done:
|
||||
if hint != nil && depth < 8 {
|
||||
if n.leaf && found {
|
||||
hint.path[depth] = byte(index + 1)
|
||||
} else {
|
||||
hint.path[depth] = byte(index)
|
||||
}
|
||||
if hint == nil || depth >= 8 {
|
||||
return index, found
|
||||
}
|
||||
|
||||
path_match:
|
||||
hint.used[depth] = true
|
||||
if n.leaf && found {
|
||||
hint.path[depth] = byte(index + 1)
|
||||
} else {
|
||||
hint.path[depth] = byte(index)
|
||||
}
|
||||
return index, found
|
||||
}
|
||||
@ -113,10 +148,10 @@ func (tr *BTree) SetHint(item interface{}, hint *PathHint) (prev interface{}) {
|
||||
if item == nil {
|
||||
panic("nil item")
|
||||
}
|
||||
tr.mu.Lock()
|
||||
prev = tr.setHint(item, hint)
|
||||
tr.mu.Unlock()
|
||||
return prev
|
||||
if tr.lock() {
|
||||
defer tr.unlock()
|
||||
}
|
||||
return tr.setHint(item, hint)
|
||||
}
|
||||
|
||||
func (tr *BTree) setHint(item interface{}, hint *PathHint) (prev interface{}) {
|
||||
@ -124,14 +159,14 @@ func (tr *BTree) setHint(item interface{}, hint *PathHint) (prev interface{}) {
|
||||
tr.root = tr.newNode(true)
|
||||
tr.root.items[0] = item
|
||||
tr.root.numItems = 1
|
||||
tr.length = 1
|
||||
tr.root.count = 1
|
||||
tr.count = 1
|
||||
return
|
||||
}
|
||||
prev = tr.nodeSet(&tr.root, item, tr.less, hint, 0)
|
||||
if prev != nil {
|
||||
return prev
|
||||
}
|
||||
tr.lnode = nil
|
||||
if tr.root.numItems == maxItems {
|
||||
n := tr.cowLoad(&tr.root)
|
||||
right, median := tr.nodeSplit(n)
|
||||
@ -140,8 +175,9 @@ func (tr *BTree) setHint(item interface{}, hint *PathHint) (prev interface{}) {
|
||||
tr.root.items[0] = median
|
||||
tr.root.children[1] = right
|
||||
tr.root.numItems = 1
|
||||
tr.root.count = n.count + 1 + right.count
|
||||
}
|
||||
tr.length++
|
||||
tr.count++
|
||||
return prev
|
||||
}
|
||||
|
||||
@ -167,10 +203,25 @@ func (tr *BTree) nodeSplit(n *node) (right *node, median interface{}) {
|
||||
n.items[i] = nil
|
||||
}
|
||||
n.numItems = maxItems / 2
|
||||
// update counts
|
||||
n.updateCount()
|
||||
right.updateCount()
|
||||
return right, median
|
||||
}
|
||||
|
||||
//go:noinline
|
||||
func (n *node) updateCount() {
|
||||
n.count = int(n.numItems)
|
||||
if !n.leaf {
|
||||
for i := 0; i <= int(n.numItems); i++ {
|
||||
n.count += n.children[i].count
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// This operation should not be inlined because it's expensive and rarely
|
||||
// called outside of heavy copy-on-write situations. Marking it "noinline"
|
||||
// allows for the parent cowLoad to be inlined.
|
||||
// go:noinline
|
||||
func (tr *BTree) copy(n *node) *node {
|
||||
n2 := *n
|
||||
n2.cow = tr.cow
|
||||
@ -182,7 +233,7 @@ func (tr *BTree) copy(n *node) *node {
|
||||
return &n2
|
||||
}
|
||||
|
||||
// cowLoad loaded the provide node and, if needed, performs a copy-on-write.
|
||||
// cowLoad loads the provide node and, if needed, performs a copy-on-write.
|
||||
func (tr *BTree) cowLoad(cn **node) *node {
|
||||
if (*cn).cow != tr.cow {
|
||||
*cn = tr.copy(*cn)
|
||||
@ -204,6 +255,7 @@ func (tr *BTree) nodeSet(cn **node, item interface{},
|
||||
copy(n.items[i+1:n.numItems+1], n.items[i:n.numItems])
|
||||
n.items[i] = item
|
||||
n.numItems++
|
||||
n.count++
|
||||
return nil
|
||||
}
|
||||
prev = tr.nodeSet(&n.children[i], item, less, hint, depth+1)
|
||||
@ -218,7 +270,8 @@ func (tr *BTree) nodeSet(cn **node, item interface{},
|
||||
n.children[i+1] = right
|
||||
n.numItems++
|
||||
}
|
||||
return prev
|
||||
n.count++
|
||||
return nil
|
||||
}
|
||||
|
||||
func (n *node) scan(iter func(item interface{}) bool) bool {
|
||||
@ -248,8 +301,9 @@ func (tr *BTree) Get(key interface{}) interface{} {
|
||||
|
||||
// GetHint gets a value for key using a path hint
|
||||
func (tr *BTree) GetHint(key interface{}, hint *PathHint) interface{} {
|
||||
tr.mu.RLock()
|
||||
defer tr.mu.RUnlock()
|
||||
if tr.rlock() {
|
||||
defer tr.runlock()
|
||||
}
|
||||
if tr.root == nil || key == nil {
|
||||
return nil
|
||||
}
|
||||
@ -270,7 +324,7 @@ func (tr *BTree) GetHint(key interface{}, hint *PathHint) interface{} {
|
||||
|
||||
// Len returns the number of items in the tree
|
||||
func (tr *BTree) Len() int {
|
||||
return tr.length
|
||||
return tr.count
|
||||
}
|
||||
|
||||
// Delete a value for a key
|
||||
@ -280,10 +334,10 @@ func (tr *BTree) Delete(key interface{}) interface{} {
|
||||
|
||||
// DeleteHint deletes a value for a key using a path hint
|
||||
func (tr *BTree) DeleteHint(key interface{}, hint *PathHint) interface{} {
|
||||
tr.mu.Lock()
|
||||
prev := tr.deleteHint(key, hint)
|
||||
tr.mu.Unlock()
|
||||
return prev
|
||||
if tr.lock() {
|
||||
defer tr.unlock()
|
||||
}
|
||||
return tr.deleteHint(key, hint)
|
||||
}
|
||||
|
||||
func (tr *BTree) deleteHint(key interface{}, hint *PathHint) interface{} {
|
||||
@ -294,12 +348,11 @@ func (tr *BTree) deleteHint(key interface{}, hint *PathHint) interface{} {
|
||||
if prev == nil {
|
||||
return nil
|
||||
}
|
||||
tr.lnode = nil
|
||||
if tr.root.numItems == 0 && !tr.root.leaf {
|
||||
tr.root = tr.root.children[0]
|
||||
}
|
||||
tr.length--
|
||||
if tr.length == 0 {
|
||||
tr.count--
|
||||
if tr.count == 0 {
|
||||
tr.root = nil
|
||||
}
|
||||
return prev
|
||||
@ -323,6 +376,7 @@ func (tr *BTree) delete(cn **node, max bool, key interface{},
|
||||
copy(n.items[i:], n.items[i+1:n.numItems])
|
||||
n.items[n.numItems-1] = nil
|
||||
n.numItems--
|
||||
n.count--
|
||||
return prev
|
||||
}
|
||||
return nil
|
||||
@ -344,6 +398,7 @@ func (tr *BTree) delete(cn **node, max bool, key interface{},
|
||||
if prev == nil {
|
||||
return nil
|
||||
}
|
||||
n.count--
|
||||
if n.children[i].numItems >= minItems {
|
||||
return prev
|
||||
}
|
||||
@ -364,10 +419,11 @@ func (tr *BTree) delete(cn **node, max bool, key interface{},
|
||||
n.children[i+1].children[:n.children[i+1].numItems+1])
|
||||
}
|
||||
n.children[i].numItems += n.children[i+1].numItems + 1
|
||||
n.children[i].count += n.children[i+1].count + 1
|
||||
copy(n.items[i:], n.items[i+1:n.numItems])
|
||||
copy(n.children[i+1:], n.children[i+2:n.numItems+1])
|
||||
n.items[n.numItems] = nil
|
||||
n.children[n.numItems+1] = nil
|
||||
n.items[n.numItems-1] = nil
|
||||
n.children[n.numItems] = nil
|
||||
n.numItems--
|
||||
} else if n.children[i].numItems > n.children[i+1].numItems {
|
||||
// move left -> right
|
||||
@ -381,30 +437,42 @@ func (tr *BTree) delete(cn **node, max bool, key interface{},
|
||||
if !n.children[0].leaf {
|
||||
n.children[i+1].children[0] =
|
||||
n.children[i].children[n.children[i].numItems]
|
||||
n.children[i+1].count += n.children[i+1].children[0].count
|
||||
}
|
||||
n.children[i+1].numItems++
|
||||
n.children[i+1].count++
|
||||
n.items[i] = n.children[i].items[n.children[i].numItems-1]
|
||||
n.children[i].items[n.children[i].numItems-1] = nil
|
||||
if !n.children[0].leaf {
|
||||
n.children[i].children[n.children[i].numItems] = nil
|
||||
n.children[i].count -= n.children[i+1].children[0].count
|
||||
}
|
||||
n.children[i].numItems--
|
||||
n.children[i].count--
|
||||
} else {
|
||||
// move right -> left
|
||||
// move left <- right
|
||||
n.children[i].items[n.children[i].numItems] = n.items[i]
|
||||
if !n.children[0].leaf {
|
||||
n.children[i].children[n.children[i].numItems+1] =
|
||||
n.children[i+1].children[0]
|
||||
n.children[i].count +=
|
||||
n.children[i].children[n.children[i].numItems+1].count
|
||||
}
|
||||
n.children[i].numItems++
|
||||
n.children[i].count++
|
||||
n.items[i] = n.children[i+1].items[0]
|
||||
copy(n.children[i+1].items[:],
|
||||
n.children[i+1].items[1:n.children[i+1].numItems])
|
||||
n.children[i+1].items[n.children[i+1].numItems-1] = nil
|
||||
if !n.children[0].leaf {
|
||||
copy(n.children[i+1].children[:],
|
||||
n.children[i+1].children[1:n.children[i+1].numItems+1])
|
||||
n.children[i+1].children[n.children[i+1].numItems] = nil
|
||||
n.children[i+1].count -=
|
||||
n.children[i].children[n.children[i].numItems].count
|
||||
}
|
||||
n.children[i+1].numItems--
|
||||
n.children[i+1].count--
|
||||
}
|
||||
return prev
|
||||
}
|
||||
@ -413,8 +481,9 @@ func (tr *BTree) delete(cn **node, max bool, key interface{},
|
||||
// Pass nil for pivot to scan all item in ascending order
|
||||
// Return false to stop iterating
|
||||
func (tr *BTree) Ascend(pivot interface{}, iter func(item interface{}) bool) {
|
||||
tr.mu.RLock()
|
||||
defer tr.mu.RUnlock()
|
||||
if tr.rlock() {
|
||||
defer tr.runlock()
|
||||
}
|
||||
if tr.root == nil {
|
||||
return
|
||||
}
|
||||
@ -476,8 +545,9 @@ func (n *node) reverse(iter func(item interface{}) bool) bool {
|
||||
// Pass nil for pivot to scan all item in descending order
|
||||
// Return false to stop iterating
|
||||
func (tr *BTree) Descend(pivot interface{}, iter func(item interface{}) bool) {
|
||||
tr.mu.RLock()
|
||||
defer tr.mu.RUnlock()
|
||||
if tr.rlock() {
|
||||
defer tr.runlock()
|
||||
}
|
||||
if tr.root == nil {
|
||||
return
|
||||
}
|
||||
@ -518,40 +588,46 @@ func (tr *BTree) Load(item interface{}) interface{} {
|
||||
if item == nil {
|
||||
panic("nil item")
|
||||
}
|
||||
tr.mu.Lock()
|
||||
defer tr.mu.Unlock()
|
||||
|
||||
// Load does not need a cowGrid because the Copy operation sets the
|
||||
// lnode to nil.
|
||||
|
||||
if tr.lnode != nil && tr.lnode.numItems < maxItems-2 {
|
||||
if tr.less(tr.lnode.items[tr.lnode.numItems-1], item) {
|
||||
tr.lnode.items[tr.lnode.numItems] = item
|
||||
tr.lnode.numItems++
|
||||
tr.length++
|
||||
return nil
|
||||
}
|
||||
if tr.lock() {
|
||||
defer tr.unlock()
|
||||
}
|
||||
prev := tr.setHint(item, nil)
|
||||
if prev != nil {
|
||||
return prev
|
||||
if tr.root == nil {
|
||||
return tr.setHint(item, nil)
|
||||
}
|
||||
n := tr.root
|
||||
n := tr.cowLoad(&tr.root)
|
||||
for {
|
||||
n.count++ // optimistically update counts
|
||||
if n.leaf {
|
||||
if n.numItems < maxItems-2 {
|
||||
if tr.less(n.items[n.numItems-1], item) {
|
||||
n.items[n.numItems] = item
|
||||
n.numItems++
|
||||
tr.count++
|
||||
return nil
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
n = tr.cowLoad(&n.children[n.numItems])
|
||||
}
|
||||
// revert the counts
|
||||
n = tr.root
|
||||
for {
|
||||
n.count--
|
||||
if n.leaf {
|
||||
tr.lnode = n
|
||||
break
|
||||
}
|
||||
n = n.children[n.numItems]
|
||||
}
|
||||
return nil
|
||||
return tr.setHint(item, nil)
|
||||
}
|
||||
|
||||
// Min returns the minimum item in tree.
|
||||
// Returns nil if the tree has no items.
|
||||
func (tr *BTree) Min() interface{} {
|
||||
tr.mu.RLock()
|
||||
defer tr.mu.RUnlock()
|
||||
if tr.rlock() {
|
||||
defer tr.runlock()
|
||||
}
|
||||
if tr.root == nil {
|
||||
return nil
|
||||
}
|
||||
@ -567,8 +643,9 @@ func (tr *BTree) Min() interface{} {
|
||||
// Max returns the maximum item in tree.
|
||||
// Returns nil if the tree has no items.
|
||||
func (tr *BTree) Max() interface{} {
|
||||
tr.mu.RLock()
|
||||
defer tr.mu.RUnlock()
|
||||
if tr.rlock() {
|
||||
defer tr.runlock()
|
||||
}
|
||||
if tr.root == nil {
|
||||
return nil
|
||||
}
|
||||
@ -584,65 +661,179 @@ func (tr *BTree) Max() interface{} {
|
||||
// PopMin removes the minimum item in tree and returns it.
|
||||
// Returns nil if the tree has no items.
|
||||
func (tr *BTree) PopMin() interface{} {
|
||||
tr.mu.Lock()
|
||||
defer tr.mu.Unlock()
|
||||
if tr.lock() {
|
||||
defer tr.unlock()
|
||||
}
|
||||
if tr.root == nil {
|
||||
return nil
|
||||
}
|
||||
tr.lnode = nil
|
||||
n := tr.cowLoad(&tr.root)
|
||||
var item interface{}
|
||||
for {
|
||||
n.count-- // optimistically update counts
|
||||
if n.leaf {
|
||||
item := n.items[0]
|
||||
item = n.items[0]
|
||||
if n.numItems == minItems {
|
||||
return tr.deleteHint(item, nil)
|
||||
break
|
||||
}
|
||||
copy(n.items[:], n.items[1:])
|
||||
n.items[n.numItems-1] = nil
|
||||
n.numItems--
|
||||
tr.length--
|
||||
if tr.length == 0 {
|
||||
tr.count--
|
||||
if tr.count == 0 {
|
||||
tr.root = nil
|
||||
}
|
||||
return item
|
||||
}
|
||||
n = tr.cowLoad(&n.children[0])
|
||||
}
|
||||
// revert the counts
|
||||
n = tr.root
|
||||
for {
|
||||
n.count++
|
||||
if n.leaf {
|
||||
break
|
||||
}
|
||||
n = n.children[0]
|
||||
}
|
||||
return tr.deleteHint(item, nil)
|
||||
}
|
||||
|
||||
// PopMax removes the minimum item in tree and returns it.
|
||||
// Returns nil if the tree has no items.
|
||||
func (tr *BTree) PopMax() interface{} {
|
||||
tr.mu.Lock()
|
||||
defer tr.mu.Unlock()
|
||||
if tr.lock() {
|
||||
defer tr.unlock()
|
||||
}
|
||||
if tr.root == nil {
|
||||
return nil
|
||||
}
|
||||
tr.lnode = nil
|
||||
n := tr.cowLoad(&tr.root)
|
||||
var item interface{}
|
||||
for {
|
||||
n.count-- // optimistically update counts
|
||||
if n.leaf {
|
||||
item := n.items[n.numItems-1]
|
||||
item = n.items[n.numItems-1]
|
||||
if n.numItems == minItems {
|
||||
return tr.deleteHint(item, nil)
|
||||
break
|
||||
}
|
||||
n.items[n.numItems-1] = nil
|
||||
n.numItems--
|
||||
tr.length--
|
||||
if tr.length == 0 {
|
||||
tr.count--
|
||||
if tr.count == 0 {
|
||||
tr.root = nil
|
||||
}
|
||||
return item
|
||||
}
|
||||
n = tr.cowLoad(&n.children[n.numItems])
|
||||
}
|
||||
// revert the counts
|
||||
n = tr.root
|
||||
for {
|
||||
n.count++
|
||||
if n.leaf {
|
||||
break
|
||||
}
|
||||
n = n.children[n.numItems]
|
||||
}
|
||||
return tr.deleteHint(item, nil)
|
||||
}
|
||||
|
||||
// GetAt returns the value at index.
|
||||
// Return nil if the tree is empty or the index is out of bounds.
|
||||
func (tr *BTree) GetAt(index int) interface{} {
|
||||
if tr.rlock() {
|
||||
defer tr.runlock()
|
||||
}
|
||||
if tr.root == nil || index < 0 || index >= tr.count {
|
||||
return nil
|
||||
}
|
||||
n := tr.root
|
||||
for {
|
||||
if n.leaf {
|
||||
return n.items[index]
|
||||
}
|
||||
i := 0
|
||||
for ; i < int(n.numItems); i++ {
|
||||
if index < n.children[i].count {
|
||||
break
|
||||
} else if index == n.children[i].count {
|
||||
return n.items[i]
|
||||
}
|
||||
index -= n.children[i].count + 1
|
||||
}
|
||||
n = n.children[i]
|
||||
}
|
||||
}
|
||||
|
||||
// DeleteAt deletes the item at index.
|
||||
// Return nil if the tree is empty or the index is out of bounds.
|
||||
func (tr *BTree) DeleteAt(index int) interface{} {
|
||||
if tr.lock() {
|
||||
defer tr.unlock()
|
||||
}
|
||||
if tr.root == nil || index < 0 || index >= tr.count {
|
||||
return nil
|
||||
}
|
||||
var pathbuf [8]uint8 // track the path
|
||||
path := pathbuf[:0]
|
||||
var item interface{}
|
||||
n := tr.cowLoad(&tr.root)
|
||||
outer:
|
||||
for {
|
||||
n.count-- // optimistically update counts
|
||||
if n.leaf {
|
||||
// the index is the item position
|
||||
item = n.items[index]
|
||||
if n.numItems == minItems {
|
||||
path = append(path, uint8(index))
|
||||
break outer
|
||||
}
|
||||
copy(n.items[index:], n.items[index+1:n.numItems])
|
||||
n.items[n.numItems-1] = nil
|
||||
n.numItems--
|
||||
tr.count--
|
||||
if tr.count == 0 {
|
||||
tr.root = nil
|
||||
}
|
||||
return item
|
||||
}
|
||||
i := 0
|
||||
for ; i < int(n.numItems); i++ {
|
||||
if index < n.children[i].count {
|
||||
break
|
||||
} else if index == n.children[i].count {
|
||||
item = n.items[i]
|
||||
path = append(path, uint8(i))
|
||||
break outer
|
||||
}
|
||||
index -= n.children[i].count + 1
|
||||
}
|
||||
path = append(path, uint8(i))
|
||||
n = tr.cowLoad(&n.children[i])
|
||||
}
|
||||
// revert the counts
|
||||
var hint PathHint
|
||||
n = tr.root
|
||||
for i := 0; i < len(path); i++ {
|
||||
if i < len(hint.path) {
|
||||
hint.path[i] = path[i]
|
||||
hint.used[i] = true
|
||||
}
|
||||
n.count++
|
||||
if !n.leaf {
|
||||
n = n.children[uint8(path[i])]
|
||||
}
|
||||
}
|
||||
return tr.deleteHint(item, &hint)
|
||||
}
|
||||
|
||||
// Height returns the height of the tree.
|
||||
// Returns zero if tree has no items.
|
||||
func (tr *BTree) Height() int {
|
||||
tr.mu.RLock()
|
||||
defer tr.mu.RUnlock()
|
||||
if tr.rlock() {
|
||||
defer tr.runlock()
|
||||
}
|
||||
var height int
|
||||
if tr.root != nil {
|
||||
n := tr.root
|
||||
@ -660,8 +851,9 @@ func (tr *BTree) Height() int {
|
||||
// Walk iterates over all items in tree, in order.
|
||||
// The items param will contain one or more items.
|
||||
func (tr *BTree) Walk(iter func(item []interface{})) {
|
||||
tr.mu.RLock()
|
||||
defer tr.mu.RUnlock()
|
||||
if tr.rlock() {
|
||||
defer tr.runlock()
|
||||
}
|
||||
if tr.root != nil {
|
||||
tr.root.walk(iter)
|
||||
}
|
||||
@ -682,12 +874,34 @@ func (n *node) walk(iter func(item []interface{})) {
|
||||
// Copy the tree. This operation is very fast because it only performs a
|
||||
// shadowed copy.
|
||||
func (tr *BTree) Copy() *BTree {
|
||||
tr.mu.Lock()
|
||||
tr.lnode = nil
|
||||
if tr.lock() {
|
||||
defer tr.unlock()
|
||||
}
|
||||
tr.cow = new(cow)
|
||||
tr2 := *tr
|
||||
tr2.mu = new(sync.RWMutex)
|
||||
tr2.cow = new(cow)
|
||||
tr.mu.Unlock()
|
||||
return &tr2
|
||||
}
|
||||
|
||||
func (tr *BTree) lock() bool {
|
||||
if tr.locks {
|
||||
tr.mu.Lock()
|
||||
}
|
||||
return tr.locks
|
||||
}
|
||||
|
||||
func (tr *BTree) unlock() {
|
||||
tr.mu.Unlock()
|
||||
}
|
||||
|
||||
func (tr *BTree) rlock() bool {
|
||||
if tr.locks {
|
||||
tr.mu.RLock()
|
||||
}
|
||||
return tr.locks
|
||||
}
|
||||
|
||||
func (tr *BTree) runlock() {
|
||||
tr.mu.RUnlock()
|
||||
}
|
||||
|
3
vendor/github.com/tidwall/btree/go.mod
generated
vendored
Normal file
3
vendor/github.com/tidwall/btree/go.mod
generated
vendored
Normal file
@ -0,0 +1,3 @@
|
||||
module github.com/tidwall/btree
|
||||
|
||||
go 1.16
|
7
vendor/github.com/tidwall/buntdb/README.md
generated
vendored
7
vendor/github.com/tidwall/buntdb/README.md
generated
vendored
@ -3,7 +3,6 @@
|
||||
src="logo.png"
|
||||
width="307" height="150" border="0" alt="BuntDB">
|
||||
<br>
|
||||
<a href="https://goreportcard.com/report/github.com/tidwall/buntdb"><img src="https://goreportcard.com/badge/github.com/tidwall/buntdb?style=flat-square" alt="Go Report Card"></a>
|
||||
<a href="https://godoc.org/github.com/tidwall/buntdb"><img src="https://img.shields.io/badge/go-documentation-blue.svg?style=flat-square" alt="Godoc"></a>
|
||||
<a href="https://github.com/tidwall/buntdb/blob/master/LICENSE"><img src="https://img.shields.io/github/license/tidwall/buntdb.svg?style=flat-square" alt="LICENSE"></a>
|
||||
</p>
|
||||
@ -28,7 +27,6 @@ Features
|
||||
- Flexible [iteration](#iterating) of data; ascending, descending, and ranges
|
||||
- [Durable append-only file](#append-only-file) format for persistence
|
||||
- Option to evict old items with an [expiration](#data-expiration) TTL
|
||||
- Tight codebase, under 2K loc using the `cloc` command
|
||||
- ACID semantics with locking [transactions](#transactions) that support rollbacks
|
||||
|
||||
|
||||
@ -457,8 +455,9 @@ Any index can be put in descending order by wrapping it's less function with `bu
|
||||
|
||||
```go
|
||||
db.CreateIndex("last_name_age", "*",
|
||||
buntdb.IndexJSON("name.last"),
|
||||
buntdb.Desc(buntdb.IndexJSON("age")))
|
||||
buntdb.IndexJSON("name.last"),
|
||||
buntdb.Desc(buntdb.IndexJSON("age")),
|
||||
)
|
||||
```
|
||||
|
||||
This will create a multi value index where the last name is ascending and the age is descending.
|
||||
|
110
vendor/github.com/tidwall/buntdb/buntdb.go
generated
vendored
110
vendor/github.com/tidwall/buntdb/buntdb.go
generated
vendored
@ -69,6 +69,7 @@ type DB struct {
|
||||
keys *btree.BTree // a tree of all item ordered by key
|
||||
exps *btree.BTree // a tree of items ordered by expiration
|
||||
idxs map[string]*index // the index trees.
|
||||
insIdxs []*index // a reuse buffer for gathering indexes
|
||||
flushes int // a count of the number of disk flushes
|
||||
closed bool // set when the database has been closed
|
||||
config Config // the database configuration
|
||||
@ -139,8 +140,8 @@ type exctx struct {
|
||||
func Open(path string) (*DB, error) {
|
||||
db := &DB{}
|
||||
// initialize trees and indexes
|
||||
db.keys = btree.New(lessCtx(nil))
|
||||
db.exps = btree.New(lessCtx(&exctx{db}))
|
||||
db.keys = btreeNew(lessCtx(nil))
|
||||
db.exps = btreeNew(lessCtx(&exctx{db}))
|
||||
db.idxs = make(map[string]*index)
|
||||
// initialize default configuration
|
||||
db.config = Config{
|
||||
@ -200,10 +201,11 @@ func (db *DB) Save(wr io.Writer) error {
|
||||
defer db.mu.RUnlock()
|
||||
// use a buffered writer and flush every 4MB
|
||||
var buf []byte
|
||||
now := time.Now()
|
||||
// iterated through every item in the database and write to the buffer
|
||||
btreeAscend(db.keys, func(item interface{}) bool {
|
||||
dbi := item.(*dbItem)
|
||||
buf = dbi.writeSetTo(buf)
|
||||
buf = dbi.writeSetTo(buf, now)
|
||||
if len(buf) > 1024*1024*4 {
|
||||
// flush when buffer is over 4MB
|
||||
_, err = wr.Write(buf)
|
||||
@ -283,7 +285,7 @@ func (idx *index) clearCopy() *index {
|
||||
}
|
||||
// initialize with empty trees
|
||||
if nidx.less != nil {
|
||||
nidx.btr = btree.New(lessCtx(nidx))
|
||||
nidx.btr = btreeNew(lessCtx(nidx))
|
||||
}
|
||||
if nidx.rect != nil {
|
||||
nidx.rtr = rtred.New(nidx)
|
||||
@ -295,7 +297,7 @@ func (idx *index) clearCopy() *index {
|
||||
func (idx *index) rebuild() {
|
||||
// initialize trees
|
||||
if idx.less != nil {
|
||||
idx.btr = btree.New(lessCtx(idx))
|
||||
idx.btr = btreeNew(lessCtx(idx))
|
||||
}
|
||||
if idx.rect != nil {
|
||||
idx.rtr = rtred.New(idx)
|
||||
@ -454,16 +456,23 @@ func (db *DB) SetConfig(config Config) error {
|
||||
// will be replaced with the new one, and return the previous item.
|
||||
func (db *DB) insertIntoDatabase(item *dbItem) *dbItem {
|
||||
var pdbi *dbItem
|
||||
// Generate a list of indexes that this item will be inserted in to.
|
||||
idxs := db.insIdxs
|
||||
for _, idx := range db.idxs {
|
||||
if idx.match(item.key) {
|
||||
idxs = append(idxs, idx)
|
||||
}
|
||||
}
|
||||
prev := db.keys.Set(item)
|
||||
if prev != nil {
|
||||
// A previous item was removed from the keys tree. Let's
|
||||
// fully delete this item from all indexes.
|
||||
pdbi = prev.(*dbItem)
|
||||
if pdbi.opts != nil && pdbi.opts.ex {
|
||||
// Remove it from the exipres tree.
|
||||
// Remove it from the expires tree.
|
||||
db.exps.Delete(pdbi)
|
||||
}
|
||||
for _, idx := range db.idxs {
|
||||
for _, idx := range idxs {
|
||||
if idx.btr != nil {
|
||||
// Remove it from the btree index.
|
||||
idx.btr.Delete(pdbi)
|
||||
@ -479,10 +488,7 @@ func (db *DB) insertIntoDatabase(item *dbItem) *dbItem {
|
||||
// expires tree
|
||||
db.exps.Set(item)
|
||||
}
|
||||
for _, idx := range db.idxs {
|
||||
if !idx.match(item.key) {
|
||||
continue
|
||||
}
|
||||
for i, idx := range idxs {
|
||||
if idx.btr != nil {
|
||||
// Add new item to btree index.
|
||||
idx.btr.Set(item)
|
||||
@ -491,7 +497,11 @@ func (db *DB) insertIntoDatabase(item *dbItem) *dbItem {
|
||||
// Add new item to rtree index.
|
||||
idx.rtr.Insert(item)
|
||||
}
|
||||
// clear the index
|
||||
idxs[i] = nil
|
||||
}
|
||||
// reuse the index list slice
|
||||
db.insIdxs = idxs[:0]
|
||||
// we must return the previous item to the caller.
|
||||
return pdbi
|
||||
}
|
||||
@ -512,6 +522,9 @@ func (db *DB) deleteFromDatabase(item *dbItem) *dbItem {
|
||||
db.exps.Delete(pdbi)
|
||||
}
|
||||
for _, idx := range db.idxs {
|
||||
if !idx.match(pdbi.key) {
|
||||
continue
|
||||
}
|
||||
if idx.btr != nil {
|
||||
// Remove it from the btree index.
|
||||
idx.btr.Delete(pdbi)
|
||||
@ -672,6 +685,7 @@ func (db *DB) Shrink() error {
|
||||
}
|
||||
done = true
|
||||
var n int
|
||||
now := time.Now()
|
||||
btreeAscendGreaterOrEqual(db.keys, &dbItem{key: pivot},
|
||||
func(item interface{}) bool {
|
||||
dbi := item.(*dbItem)
|
||||
@ -681,7 +695,7 @@ func (db *DB) Shrink() error {
|
||||
done = false
|
||||
return false
|
||||
}
|
||||
buf = dbi.writeSetTo(buf)
|
||||
buf = dbi.writeSetTo(buf, now)
|
||||
n++
|
||||
return true
|
||||
},
|
||||
@ -908,8 +922,8 @@ func (db *DB) readLoad(rd io.Reader, modTime time.Time) (n int64, err error) {
|
||||
db.deleteFromDatabase(&dbItem{key: parts[1]})
|
||||
} else if (parts[0][0] == 'f' || parts[0][0] == 'F') &&
|
||||
strings.ToLower(parts[0]) == "flushdb" {
|
||||
db.keys = btree.New(lessCtx(nil))
|
||||
db.exps = btree.New(lessCtx(&exctx{db}))
|
||||
db.keys = btreeNew(lessCtx(nil))
|
||||
db.exps = btreeNew(lessCtx(&exctx{db}))
|
||||
db.idxs = make(map[string]*index)
|
||||
} else {
|
||||
return totalSize, ErrInvalid
|
||||
@ -941,11 +955,16 @@ func (db *DB) load() error {
|
||||
return err
|
||||
}
|
||||
}
|
||||
pos, err := db.file.Seek(n, 0)
|
||||
if err != nil {
|
||||
if _, err := db.file.Seek(n, 0); err != nil {
|
||||
return err
|
||||
}
|
||||
db.lastaofsz = int(pos)
|
||||
var estaofsz int
|
||||
db.keys.Walk(func(items []interface{}) {
|
||||
for _, v := range items {
|
||||
estaofsz += v.(*dbItem).estAOFSetSize()
|
||||
}
|
||||
})
|
||||
db.lastaofsz += estaofsz
|
||||
return nil
|
||||
}
|
||||
|
||||
@ -1054,8 +1073,8 @@ func (tx *Tx) DeleteAll() error {
|
||||
}
|
||||
|
||||
// now reset the live database trees
|
||||
tx.db.keys = btree.New(lessCtx(nil))
|
||||
tx.db.exps = btree.New(lessCtx(&exctx{tx.db}))
|
||||
tx.db.keys = btreeNew(lessCtx(nil))
|
||||
tx.db.exps = btreeNew(lessCtx(&exctx{tx.db}))
|
||||
tx.db.idxs = make(map[string]*index)
|
||||
|
||||
// finally re-create the indexes
|
||||
@ -1165,12 +1184,13 @@ func (tx *Tx) Commit() error {
|
||||
if tx.wc.rbkeys != nil {
|
||||
tx.db.buf = append(tx.db.buf, "*1\r\n$7\r\nflushdb\r\n"...)
|
||||
}
|
||||
now := time.Now()
|
||||
// Each committed record is written to disk
|
||||
for key, item := range tx.wc.commitItems {
|
||||
if item == nil {
|
||||
tx.db.buf = (&dbItem{key: key}).writeDeleteTo(tx.db.buf)
|
||||
} else {
|
||||
tx.db.buf = item.writeSetTo(tx.db.buf)
|
||||
tx.db.buf = item.writeSetTo(tx.db.buf, now)
|
||||
}
|
||||
}
|
||||
// Flushing the buffer only once per transaction.
|
||||
@ -1243,16 +1263,53 @@ type dbItem struct {
|
||||
keyless bool // keyless item for scanning
|
||||
}
|
||||
|
||||
func estIntSize(x int) int {
|
||||
if x == 0 {
|
||||
return 1
|
||||
}
|
||||
var n int
|
||||
for x > 0 {
|
||||
n++
|
||||
x /= 10
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func estArraySize(count int) int {
|
||||
return 1 + estIntSize(count) + 2
|
||||
}
|
||||
|
||||
func estBulkStringSize(s string) int {
|
||||
return 1 + estIntSize(len(s)) + 2 + len(s) + 2
|
||||
}
|
||||
|
||||
func (dbi *dbItem) estAOFSetSize() (n int) {
|
||||
if dbi.opts != nil && dbi.opts.ex {
|
||||
n += estArraySize(5)
|
||||
n += estBulkStringSize("set")
|
||||
n += estBulkStringSize(dbi.key)
|
||||
n += estBulkStringSize(dbi.val)
|
||||
n += estBulkStringSize("ex")
|
||||
n += estBulkStringSize("99") // estimate two byte bulk string
|
||||
} else {
|
||||
n += estArraySize(3)
|
||||
n += estBulkStringSize("set")
|
||||
n += estBulkStringSize(dbi.key)
|
||||
n += estBulkStringSize(dbi.val)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func appendArray(buf []byte, count int) []byte {
|
||||
buf = append(buf, '*')
|
||||
buf = append(buf, strconv.FormatInt(int64(count), 10)...)
|
||||
buf = strconv.AppendInt(buf, int64(count), 10)
|
||||
buf = append(buf, '\r', '\n')
|
||||
return buf
|
||||
}
|
||||
|
||||
func appendBulkString(buf []byte, s string) []byte {
|
||||
buf = append(buf, '$')
|
||||
buf = append(buf, strconv.FormatInt(int64(len(s)), 10)...)
|
||||
buf = strconv.AppendInt(buf, int64(len(s)), 10)
|
||||
buf = append(buf, '\r', '\n')
|
||||
buf = append(buf, s...)
|
||||
buf = append(buf, '\r', '\n')
|
||||
@ -1260,9 +1317,9 @@ func appendBulkString(buf []byte, s string) []byte {
|
||||
}
|
||||
|
||||
// writeSetTo writes an item as a single SET record to the a bufio Writer.
|
||||
func (dbi *dbItem) writeSetTo(buf []byte) []byte {
|
||||
func (dbi *dbItem) writeSetTo(buf []byte, now time.Time) []byte {
|
||||
if dbi.opts != nil && dbi.opts.ex {
|
||||
ex := time.Until(dbi.opts.exat) / time.Second
|
||||
ex := dbi.opts.exat.Sub(now) / time.Second
|
||||
buf = appendArray(buf, 5)
|
||||
buf = appendBulkString(buf, "set")
|
||||
buf = appendBulkString(buf, dbi.key)
|
||||
@ -2300,3 +2357,8 @@ func btreeDescendLessOrEqual(tr *btree.BTree, pivot interface{},
|
||||
) {
|
||||
tr.Descend(pivot, iter)
|
||||
}
|
||||
|
||||
func btreeNew(less func(a, b interface{}) bool) *btree.BTree {
|
||||
// Using NewNonConcurrent because we're managing our own locks.
|
||||
return btree.NewNonConcurrent(less)
|
||||
}
|
||||
|
6
vendor/github.com/tidwall/buntdb/go.mod
generated
vendored
6
vendor/github.com/tidwall/buntdb/go.mod
generated
vendored
@ -3,9 +3,9 @@ module github.com/tidwall/buntdb
|
||||
go 1.16
|
||||
|
||||
require (
|
||||
github.com/tidwall/btree v0.4.2
|
||||
github.com/tidwall/gjson v1.7.4
|
||||
github.com/tidwall/grect v0.1.1
|
||||
github.com/tidwall/btree v0.6.0
|
||||
github.com/tidwall/gjson v1.8.0
|
||||
github.com/tidwall/grect v0.1.2
|
||||
github.com/tidwall/lotsa v1.0.2
|
||||
github.com/tidwall/match v1.0.3
|
||||
github.com/tidwall/rtred v0.1.2
|
||||
|
12
vendor/github.com/tidwall/buntdb/go.sum
generated
vendored
12
vendor/github.com/tidwall/buntdb/go.sum
generated
vendored
@ -1,9 +1,9 @@
|
||||
github.com/tidwall/btree v0.4.2 h1:aLwwJlG+InuFzdAPuBf9YCAR1LvSQ9zhC5aorFPlIPs=
|
||||
github.com/tidwall/btree v0.4.2/go.mod h1:huei1BkDWJ3/sLXmO+bsCNELL+Bp2Kks9OLyQFkzvA8=
|
||||
github.com/tidwall/gjson v1.7.4 h1:19cchw8FOxkG5mdLRkGf9jqIqEyqdZhPqW60XfyFxk8=
|
||||
github.com/tidwall/gjson v1.7.4/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk=
|
||||
github.com/tidwall/grect v0.1.1 h1:+kMEkxhoqB7rniVXzMEIA66XwU07STgINqxh+qVIndY=
|
||||
github.com/tidwall/grect v0.1.1/go.mod h1:CzvbGiFbWUwiJ1JohXLb28McpyBsI00TK9Y6pDWLGRQ=
|
||||
github.com/tidwall/btree v0.6.0 h1:JLYAFGV+1gjyFi3iQbO/fupBin+Ooh7dxqVV0twJ1Bo=
|
||||
github.com/tidwall/btree v0.6.0/go.mod h1:TzIRzen6yHbibdSfK6t8QimqbUnoxUSrZfeW7Uob0q4=
|
||||
github.com/tidwall/gjson v1.8.0 h1:Qt+orfosKn0rbNTZqHYDqBrmm3UDA4KRkv70fDzG+PQ=
|
||||
github.com/tidwall/gjson v1.8.0/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk=
|
||||
github.com/tidwall/grect v0.1.2 h1:wKVeQVZhjaFCKTTlpkDe3Ex4ko3cMGW3MRKawRe8uQ4=
|
||||
github.com/tidwall/grect v0.1.2/go.mod h1:v+n4ewstPGduVJebcp5Eh2WXBJBumNzyhK8GZt4gHNw=
|
||||
github.com/tidwall/lotsa v1.0.2 h1:dNVBH5MErdaQ/xd9s769R31/n2dXavsQ0Yf4TMEHHw8=
|
||||
github.com/tidwall/lotsa v1.0.2/go.mod h1:X6NiU+4yHA3fE3Puvpnn1XMDrFZrE9JO2/w+UMuqgR8=
|
||||
github.com/tidwall/match v1.0.3 h1:FQUVvBImDutD8wJLN6c5eMzWtjgONK9MwIBCOrUJKeE=
|
||||
|
10
vendor/github.com/tidwall/gjson/README.md
generated
vendored
10
vendor/github.com/tidwall/gjson/README.md
generated
vendored
@ -14,8 +14,6 @@ It has features such as [one line retrieval](#get-a-value), [dot notation paths]
|
||||
|
||||
Also check out [SJSON](https://github.com/tidwall/sjson) for modifying json, and the [JJ](https://github.com/tidwall/jj) command line tool.
|
||||
|
||||
For the Rust version go to [gjson.rs](https://github.com/tidwall/gjson.rs).
|
||||
|
||||
Getting Started
|
||||
===============
|
||||
|
||||
@ -152,10 +150,6 @@ result.Less(token Result, caseSensitive bool) bool
|
||||
|
||||
The `result.Value()` function returns an `interface{}` which requires type assertion and is one of the following Go types:
|
||||
|
||||
The `result.Array()` function returns back an array of values.
|
||||
If the result represents a non-existent value, then an empty array will be returned.
|
||||
If the result is not a JSON array, the return value will be an array containing one result.
|
||||
|
||||
```go
|
||||
boolean >> bool
|
||||
number >> float64
|
||||
@ -165,6 +159,10 @@ array >> []interface{}
|
||||
object >> map[string]interface{}
|
||||
```
|
||||
|
||||
The `result.Array()` function returns back an array of values.
|
||||
If the result represents a non-existent value, then an empty array will be returned.
|
||||
If the result is not a JSON array, the return value will be an array containing one result.
|
||||
|
||||
### 64-bit integers
|
||||
|
||||
The `result.Int()` and `result.Uint()` calls are capable of reading all 64 bits, allowing for large JSON integers.
|
||||
|
152
vendor/github.com/tidwall/gjson/gjson.go
generated
vendored
152
vendor/github.com/tidwall/gjson/gjson.go
generated
vendored
@ -714,10 +714,10 @@ type arrayPathResult struct {
|
||||
alogkey string
|
||||
query struct {
|
||||
on bool
|
||||
all bool
|
||||
path string
|
||||
op string
|
||||
value string
|
||||
all bool
|
||||
}
|
||||
}
|
||||
|
||||
@ -750,120 +750,27 @@ func parseArrayPath(path string) (r arrayPathResult) {
|
||||
} else if path[1] == '[' || path[1] == '(' {
|
||||
// query
|
||||
r.query.on = true
|
||||
if true {
|
||||
qpath, op, value, _, fi, ok := parseQuery(path[i:])
|
||||
if !ok {
|
||||
// bad query, end now
|
||||
break
|
||||
}
|
||||
r.query.path = qpath
|
||||
r.query.op = op
|
||||
r.query.value = value
|
||||
i = fi - 1
|
||||
if i+1 < len(path) && path[i+1] == '#' {
|
||||
r.query.all = true
|
||||
}
|
||||
} else {
|
||||
var end byte
|
||||
if path[1] == '[' {
|
||||
end = ']'
|
||||
} else {
|
||||
end = ')'
|
||||
}
|
||||
i += 2
|
||||
// whitespace
|
||||
for ; i < len(path); i++ {
|
||||
if path[i] > ' ' {
|
||||
break
|
||||
}
|
||||
}
|
||||
s := i
|
||||
for ; i < len(path); i++ {
|
||||
if path[i] <= ' ' ||
|
||||
path[i] == '!' ||
|
||||
path[i] == '=' ||
|
||||
path[i] == '<' ||
|
||||
path[i] == '>' ||
|
||||
path[i] == '%' ||
|
||||
path[i] == end {
|
||||
break
|
||||
}
|
||||
}
|
||||
r.query.path = path[s:i]
|
||||
// whitespace
|
||||
for ; i < len(path); i++ {
|
||||
if path[i] > ' ' {
|
||||
break
|
||||
}
|
||||
}
|
||||
if i < len(path) {
|
||||
s = i
|
||||
if path[i] == '!' {
|
||||
if i < len(path)-1 && (path[i+1] == '=' ||
|
||||
path[i+1] == '%') {
|
||||
i++
|
||||
}
|
||||
} else if path[i] == '<' || path[i] == '>' {
|
||||
if i < len(path)-1 && path[i+1] == '=' {
|
||||
i++
|
||||
}
|
||||
} else if path[i] == '=' {
|
||||
if i < len(path)-1 && path[i+1] == '=' {
|
||||
s++
|
||||
i++
|
||||
}
|
||||
}
|
||||
i++
|
||||
r.query.op = path[s:i]
|
||||
// whitespace
|
||||
for ; i < len(path); i++ {
|
||||
if path[i] > ' ' {
|
||||
break
|
||||
}
|
||||
}
|
||||
s = i
|
||||
for ; i < len(path); i++ {
|
||||
if path[i] == '"' {
|
||||
i++
|
||||
s2 := i
|
||||
for ; i < len(path); i++ {
|
||||
if path[i] > '\\' {
|
||||
continue
|
||||
}
|
||||
if path[i] == '"' {
|
||||
// look for an escaped slash
|
||||
if path[i-1] == '\\' {
|
||||
n := 0
|
||||
for j := i - 2; j > s2-1; j-- {
|
||||
if path[j] != '\\' {
|
||||
break
|
||||
}
|
||||
n++
|
||||
}
|
||||
if n%2 == 0 {
|
||||
continue
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
} else if path[i] == end {
|
||||
if i+1 < len(path) && path[i+1] == '#' {
|
||||
r.query.all = true
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
if i > len(path) {
|
||||
i = len(path)
|
||||
}
|
||||
v := path[s:i]
|
||||
for len(v) > 0 && v[len(v)-1] <= ' ' {
|
||||
v = v[:len(v)-1]
|
||||
}
|
||||
r.query.value = v
|
||||
qpath, op, value, _, fi, vesc, ok :=
|
||||
parseQuery(path[i:])
|
||||
if !ok {
|
||||
// bad query, end now
|
||||
break
|
||||
}
|
||||
if len(value) > 2 && value[0] == '"' &&
|
||||
value[len(value)-1] == '"' {
|
||||
value = value[1 : len(value)-1]
|
||||
if vesc {
|
||||
value = unescape(value)
|
||||
}
|
||||
}
|
||||
r.query.path = qpath
|
||||
r.query.op = op
|
||||
r.query.value = value
|
||||
|
||||
i = fi - 1
|
||||
if i+1 < len(path) && path[i+1] == '#' {
|
||||
r.query.all = true
|
||||
}
|
||||
}
|
||||
}
|
||||
continue
|
||||
@ -889,11 +796,11 @@ func parseArrayPath(path string) (r arrayPathResult) {
|
||||
// # middle
|
||||
// .cap # right
|
||||
func parseQuery(query string) (
|
||||
path, op, value, remain string, i int, ok bool,
|
||||
path, op, value, remain string, i int, vesc, ok bool,
|
||||
) {
|
||||
if len(query) < 2 || query[0] != '#' ||
|
||||
(query[1] != '(' && query[1] != '[') {
|
||||
return "", "", "", "", i, false
|
||||
return "", "", "", "", i, false, false
|
||||
}
|
||||
i = 2
|
||||
j := 0 // start of value part
|
||||
@ -921,6 +828,7 @@ func parseQuery(query string) (
|
||||
i++
|
||||
for ; i < len(query); i++ {
|
||||
if query[i] == '\\' {
|
||||
vesc = true
|
||||
i++
|
||||
} else if query[i] == '"' {
|
||||
break
|
||||
@ -929,7 +837,7 @@ func parseQuery(query string) (
|
||||
}
|
||||
}
|
||||
if depth > 0 {
|
||||
return "", "", "", "", i, false
|
||||
return "", "", "", "", i, false, false
|
||||
}
|
||||
if j > 0 {
|
||||
path = trim(query[2:j])
|
||||
@ -966,7 +874,7 @@ func parseQuery(query string) (
|
||||
path = trim(query[2:i])
|
||||
remain = query[i+1:]
|
||||
}
|
||||
return path, op, value, remain, i + 1, true
|
||||
return path, op, value, remain, i + 1, vesc, true
|
||||
}
|
||||
|
||||
func trim(s string) string {
|
||||
@ -1266,8 +1174,14 @@ func parseObject(c *parseContext, i int, path string) (int, bool) {
|
||||
}
|
||||
func queryMatches(rp *arrayPathResult, value Result) bool {
|
||||
rpv := rp.query.value
|
||||
if len(rpv) > 2 && rpv[0] == '"' && rpv[len(rpv)-1] == '"' {
|
||||
rpv = rpv[1 : len(rpv)-1]
|
||||
if len(rpv) > 0 && rpv[0] == '~' {
|
||||
// convert to bool
|
||||
rpv = rpv[1:]
|
||||
if value.Bool() {
|
||||
value = Result{Type: True}
|
||||
} else {
|
||||
value = Result{Type: False}
|
||||
}
|
||||
}
|
||||
if !value.Exists() {
|
||||
return false
|
||||
|
2
vendor/github.com/tidwall/grect/go.mod
generated
vendored
2
vendor/github.com/tidwall/grect/go.mod
generated
vendored
@ -2,4 +2,4 @@ module github.com/tidwall/grect
|
||||
|
||||
go 1.15
|
||||
|
||||
require github.com/tidwall/gjson v1.7.4
|
||||
require github.com/tidwall/gjson v1.8.0
|
||||
|
4
vendor/github.com/tidwall/grect/go.sum
generated
vendored
4
vendor/github.com/tidwall/grect/go.sum
generated
vendored
@ -1,5 +1,5 @@
|
||||
github.com/tidwall/gjson v1.7.4 h1:19cchw8FOxkG5mdLRkGf9jqIqEyqdZhPqW60XfyFxk8=
|
||||
github.com/tidwall/gjson v1.7.4/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk=
|
||||
github.com/tidwall/gjson v1.8.0 h1:Qt+orfosKn0rbNTZqHYDqBrmm3UDA4KRkv70fDzG+PQ=
|
||||
github.com/tidwall/gjson v1.8.0/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk=
|
||||
github.com/tidwall/match v1.0.3 h1:FQUVvBImDutD8wJLN6c5eMzWtjgONK9MwIBCOrUJKeE=
|
||||
github.com/tidwall/match v1.0.3/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
|
||||
github.com/tidwall/pretty v1.1.0 h1:K3hMW5epkdAVwibsQEfR/7Zj0Qgt4DxtNumTq/VloO8=
|
||||
|
18
vendor/modules.txt
vendored
18
vendor/modules.txt
vendored
@ -7,8 +7,6 @@ github.com/GehirnInc/crypt
|
||||
github.com/GehirnInc/crypt/common
|
||||
github.com/GehirnInc/crypt/internal
|
||||
github.com/GehirnInc/crypt/md5_crypt
|
||||
# github.com/dgrijalva/jwt-go v3.2.0+incompatible
|
||||
## explicit
|
||||
# github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815
|
||||
## explicit
|
||||
github.com/docopt/docopt-go
|
||||
@ -35,8 +33,6 @@ github.com/golang-jwt/jwt
|
||||
# github.com/gorilla/websocket v1.4.2 => github.com/ergochat/websocket v1.4.2-oragono1
|
||||
## explicit
|
||||
github.com/gorilla/websocket
|
||||
# github.com/goshuirc/irc-go v0.0.0-20210318074529-bdc2c2cd2fef
|
||||
## explicit
|
||||
# github.com/okzk/sdnotify v0.0.0-20180710141335-d9becc38acbd
|
||||
## explicit
|
||||
github.com/okzk/sdnotify
|
||||
@ -44,20 +40,16 @@ github.com/okzk/sdnotify
|
||||
## explicit
|
||||
# github.com/onsi/gomega v1.9.0
|
||||
## explicit
|
||||
# github.com/oragono/confusables v0.0.0-20201108231250-4ab98ab61fb1
|
||||
## explicit
|
||||
# github.com/oragono/go-ident v0.0.0-20200511222032-830550b1d775
|
||||
## explicit
|
||||
# github.com/stretchr/testify v1.4.0
|
||||
## explicit
|
||||
# github.com/tidwall/btree v0.4.2
|
||||
# github.com/tidwall/btree v0.6.0
|
||||
github.com/tidwall/btree
|
||||
# github.com/tidwall/buntdb v1.2.3
|
||||
# github.com/tidwall/buntdb v1.2.6
|
||||
## explicit
|
||||
github.com/tidwall/buntdb
|
||||
# github.com/tidwall/gjson v1.7.4
|
||||
# github.com/tidwall/gjson v1.8.0
|
||||
github.com/tidwall/gjson
|
||||
# github.com/tidwall/grect v0.1.1
|
||||
# github.com/tidwall/grect v0.1.2
|
||||
github.com/tidwall/grect
|
||||
# github.com/tidwall/match v1.0.3
|
||||
github.com/tidwall/match
|
||||
@ -66,8 +58,6 @@ github.com/tidwall/pretty
|
||||
# github.com/tidwall/rtred v0.1.2
|
||||
github.com/tidwall/rtred
|
||||
github.com/tidwall/rtred/base
|
||||
# github.com/tidwall/rtree v0.0.0-20201027154624-32188eeb08a8
|
||||
## explicit
|
||||
# github.com/tidwall/tinyqueue v0.1.1
|
||||
github.com/tidwall/tinyqueue
|
||||
# github.com/toorop/go-dkim v0.0.0-20201103131630-e1cd1a0a5208
|
||||
|
Loading…
Reference in New Issue
Block a user