* Avoid Watches using write transactions
* Prevent duplicated events
By stopping the backfilling once it reaches the latest ResourceVersion at the moment the watcher was registered
* Move function
* Add comment
* Close channel after removing the watcher
* Remove unnecessary case
* Take derived context into account
* Remove unnecessary case
* Properly wait for goroutine to finish before returning from Watch
This prevents a panic trying to write to the original eventsChan if it was closed
* Simplify buffler flush section
* Parameterize maxBufferSize
* Refactor into a single goroutine
* Split goroutine into 3 different loops for readability
* Release queue for GC once no longer needed
* Change order of primary key for _events tables
* Capture type info so numeric values sort correctly.
* Add tabular data for other metadata.fields to sort numerically.
The issue is that some fields are not numeric, but are represented
in an entry in metadata.fields with a numeric indicator. Like
with secrets, metadata.fields[2] gives the number of actual keys
in the stored secret.
* WIP - Add tests and the custom function.
* Add a converter to convert the annotated strings to float64's
* Fold the 'builtinIntTable' into the prepopulated 'typeGuidanceTable'.
* WIP - get memory-tests to pass for now
* More test tweaking - verify the status.available fields work when not specified.
* WIP - get memory-tests to pass for now
* WIP - Add the associations for pcio.clusters -> mcio.clusters.
* WIP - need more stuff. Revert this commit?
* WIP - remove these
* Fix rebase cruft
* avail -> requested, and prov is in v1, not v3
* Include cpuRaw in the injected fields.
* Make this debug output a trace so we can access it for debugging
* Start management.cattle.io.clusters caches for provisioning.cattle.io.clusters
* Add comment
---------
Co-authored-by: Eric Promislow <epromislow@suse.com>
* Capture type info so numeric values sort correctly.
* Replace conditional string prefixing with strings.TrimPrefix
* Add tabular data for other metadata.fields to sort numerically.
The issue is that some fields are not numeric, but are represented
in an entry in metadata.fields with a numeric indicator. Like
with secrets, metadata.fields[2] gives the number of actual keys
in the stored secret.
* Index more fields round 7, part 1
Add status.{allocatable,requested}.{cpus,pods,memory,memoryRaw}
* Add a converter to convert the annotated strings to float64's
* Fold the 'builtinIntTable' into the prepopulated 'typeGuidanceTable'.
- More test tweaking - verify the status.available fields work when not specified.
* Correction: 'status.available' should be 'status.requested'.
* Add the 'memoryRaw' field to status.A/R
* golint change, but the sql function uses underscores
* Fix decoding problems.
* Remove redundant type registering for GOB
* We no longer need to process this type because everything is 'map[string]any'.
* Remove commented-out debug helpers
* Use k8s library quantity-parser over ours.
Our unit tests pass as if with this change, looks good.
* SQL codegen: map the SortAsIP option to the 'inet_aton' function call
* Implement and test sorting IP addresses by their underlying int64 value.
* Implement parsing 'ip(x)' as an IP-sort directive.
* Fix a change rebasing missed.
* Capture type info so numeric values sort correctly.
* Replace conditional string prefixing with strings.TrimPrefix
* Add tabular data for other metadata.fields to sort numerically.
The issue is that some fields are not numeric, but are represented
in an entry in metadata.fields with a numeric indicator. Like
with secrets, metadata.fields[2] gives the number of actual keys
in the stored secret.
* Update added test to include the typeGuidance param.
* Act on review suggestions/comments.
* Add a unit test verifying typeGuidance can change a DB field type.
The test found an error that it's better for getFieldAndColInfo
to return default typeGuidance = an empty map rather than `nil`.
* Add Tom's unit test to verify TypeGuidance influences sorting.
Also add a fake type-guidance as int on a field where the values
start with a number but also have non-numeric data. sqlite stores
the entire string, and sort maps the values to 0... so sorting
doesn't repeats the incoming order (and sort-DESC reverses it).
* Add a clarifying comment on sorting non-ints or partly-ints as INT.
* Add more test objects to verify sorting mix of ints and non-ints works as expected.
* Revert OnSchemas change work
* Track schema changes
* Only stop a single GVK informer factory
* Add tests
* Rename crd to crdClient
* Rename s to sqlStore
* Don't wait for synced caches if request is canceled
* Move schematracker to pkg/sqlcache/schematracker
* Wait for garbage collection to be finished before creating a new DB
* Prevent DB reset while inflight List requests
* Rename cacheFor to cacheForLocked
* Rename Reset to Stop
* Mention DoneWithCache in README
* Add RunGC to ByOptionsLister interface
* Further fixes to the projectsornamespaces param, with new tests.
* Simplify projectsornamespaces on namespace kind
Move the processing of this special case from the SQL code generator
to the parser, and map it to simple `metadata.name=` and `metadata.labels[...]`
tests.
* Fix `projectsornamespace!=` code gen.
These should generate (f.namespace != X AND (negative label test).
The code was generating an `OR` giving too many false positives.
* Fix the projORns not error in the SQL
* working for positive case
* changed to custom code
* removed comment
* added tests
* fixing ci error
* one more fix
* Update proxy_store tests for projectornamespace parsing.
Since we no longer need to execute a db query, and just return a filter,
we can drop a lot of code.
* added distinct back to conform with other queries
* one more fix
---------
Co-authored-by: Eric Promislow <epromislow@suse.com>
* Sort by ID rather than metadata.namespace,metadata.name
- In tests add an id field to the list of fields to cache/create and to the objects.
* Add benchmark tests to compare sorting by ns/name vs id.
* changed ParseQuery projectornamespace handling
* addressed comments from eric/chad and tom
* fixing lint error
* added resourceVersion to list
* updated mocks
* updated tests
* Update tests after rebase
---------
Co-authored-by: Tom Lebreux <tom.lebreux@suse.com>
* More error wrapping for SQL cache
* Use context.Context instead of stopCh
* Move CacheFor to Info log level
* Implement time-based GC
* Set default GC param when running standalone steve
* Implement hard-wired external associations:
* The table is in sqlproxy.proxy_store
- externalGVKDependencies - a map of GVKs to dependencies.
When the key GVK is updated, it triggers the updates in the database for the dependent GVKs,
replacing fields as specified in the table.
* This is done in an afterUpsert handler, but it's done after the transaction for the core
GVK update is finished, because most likely the dependent GVK updates will depend on the
final database values for the GVK being updated, and if we do it as part of the transaction
the new values won't be committed to the database.
* Wrote unit tests for external associations.
* When an object is modified/created, check for external deps that need updating.
* Stop emitting errors when joining tables if one of the tables doesn't exist.
* Finish rebasing manually.
* Fix external association unit-tests after rebasing.
Specifically, recomment a test that was uncommented in this PR,
and revert the underlying code that made this test pass. That
code change causes user-based integration tests to fail, so something
is not correct in that code.
This was actually unrelated to the reason for PR 633, to fix
a problem when sorting on a label that dropped objects that
didn't have that label. That part remains untouched.
* Use SQL `WITH` statements to sort unbound labels.
These are labels whose names are never positively referenced in a filter,
so they don't need to exist on the row that we still want to display.
Here I create a virtual table of all the rows, substituting a null
value for each label that isn't associated on the row, and then sort on that.
* Just always select-distinct for now.
* Add more tests for filtering/sorting
- Assign more values to the cattle and horses labels
- Move the sortfield value to a number in the 100s -- keep in mind
these values are sorted by ascii value of underlying chars, not numerically
- Rename the var names to better reflect the values they contain
* Remove mentions of the prepared SQL statement in the test descriptions.
We care either about the items we get back from the query, or in some
cases the SQL that gets generated by the AST interpreter.
* Simplify the use of WITH stmts in SQL (thx Tom)
* Fix the comment about an unexpected situation.
* Fix post-rebase tests.
* Continue rebasing.
* Wrote unit tests for external associations.
* Fix the generated SQL.
Some syntactic sugar (capitalizing the keywords), but use the 'ON' syntax on JOINs.
* We want "management.cattle.io.projects:spec.displayName" not "...spec.clusterName"
* Implement hard-wired external associations:
* The table is in sqlproxy.proxy_store
- externalGVKDependencies - a map of GVKs to dependencies.
When the key GVK is updated, it triggers the updates in the database for the dependent GVKs,
replacing fields as specified in the table.
* This is done in an afterUpsert handler, but it's done after the transaction for the core
GVK update is finished, because most likely the dependent GVK updates will depend on the
final database values for the GVK being updated, and if we do it as part of the transaction
the new values won't be committed to the database.
* When an object is modified/created, check for external deps that need updating.
* Stop emitting errors when joining tables if one of the tables doesn't exist.
* Update unit test syntax for SQL queries.
* And an override check
This ensures we don't overwrite good data when
pulling data from one table to another.
* Drop labels, and use mgmt.cattle.io/spec.displayName
There's no need to hardwire labels in proxy_store:typeSpecificIndexedFields
because all labels are indexed in the shadow labels table.
* Keep clusterName, add displayName for mgmt.cattle.io
* Fix rebase/merge breakage.
* Finish the merge: add the 'selfUpdateInfo' param where it didn't get inserted during merge.
* Patch up rebase failures.
* Now gomock generates named args. I give up.
* Continue rebasing.
* Wrote unit tests for external associations.
* Fix the generated SQL.
Some syntactic sugar (capitalizing the keywords), but use the 'ON' syntax on JOINs.
* whitespace fix post rebase
* We want "management.cattle.io.projects:spec.displayName" not "...spec.clusterName"
* Fix the database calls: drop the key
* Fix breakage during automatic rebase merging gone wrong.
* ws fix - NFC
* Post rebase-merge fixes
* Fix rebase-driven merge.
* Fix rebase breakage.
* go-uber v0.5.2 prefers real arg names to '<argN>'
* Run tests using sqlite DB in a temp directory.
I was running into write-file errors which happens when two sqlite processes try to update the DB at the same time.
* Implement and test the extractBarredValue custom SQL function.
* Explain the DB path constants better.
* Return watch error instead of logging it
The UI needs to know about watch error like `resourceversion too old` so
we need to return it.
* Sort by resourceVersion as number
The UI makes some assumption on resourceVersion. It assumes they are a
number and they are ordered by the number value. We'll want to fix this
at some point most likely but for now let's give something in a way that
UI wants.
* Remove -d suffix
After much testing, a delete of an object seems to have its own
resourceVersion so we don't need the -d suffix, we can simply use the
new resourceVersion.
* Add ListOptionIndexerOptions to create ListOptionIndexer
* Remove unused function
* Implement configurable garbage collection on ListOptionIndexer
* Propagate Steve options from Server
* Re-order SQL event hooks so events are last
* Add QueryRowContext for single line queries
* Add test case for unknown resource version
* Properly check rows and close it
* More accurate error message when context.Context is canceled
* Re-order test check
* Split RegisterAfterUpsert into two
We're going to need to be able to differentiate between Add and Update
for storing events in the _events table.
* Update mocks
Alejandro Ruiz