Commit Graph

17 Commits

Author SHA1 Message Date
Eric Promislow e03f03ea75
Index fields: part 7 (#855)
* Capture type info so numeric values sort correctly.

* Replace conditional string prefixing with strings.TrimPrefix

* Add tabular data for other metadata.fields to sort numerically.

The issue is that some fields are not numeric, but are represented
in an entry in metadata.fields with a numeric indicator.  Like
with secrets, metadata.fields[2] gives the number of actual keys
in the stored secret.

* Index more fields round 7, part 1

Add status.{allocatable,requested}.{cpus,pods,memory,memoryRaw}

* Add a converter to convert the annotated strings to float64's

* Fold the 'builtinIntTable' into the prepopulated 'typeGuidanceTable'.

- More test tweaking - verify the status.available fields work when not specified.

* Correction: 'status.available' should be 'status.requested'.

* Add the 'memoryRaw' field to status.A/R

* golint change, but the sql function uses underscores

* Fix decoding problems.

* Remove redundant type registering for GOB

* We no longer need to process this type because everything is 'map[string]any'.

* Remove commented-out debug helpers

* Use k8s library quantity-parser over ours.

Our unit tests pass as if with this change, looks good.
2025-10-17 08:55:15 -04:00
Eric Promislow 4f27f71146
Sort by ipaddr (#760)
* SQL codegen: map the SortAsIP option to the 'inet_aton' function call

* Implement and test sorting IP addresses by their underlying int64 value.

* Implement parsing 'ip(x)' as an IP-sort directive.

* Fix a change rebasing missed.
2025-10-15 13:45:10 -07:00
Alejandro Ruiz 6e39b34488
Make vai use the encoding abstraction and improvements (#743) 2025-10-10 08:18:56 +02:00
Alejandro Ruiz b533087afe
Add debug logging for queries (#834) 2025-10-08 08:00:44 +02:00
Alejandro Ruiz fb0bb4693d
Refactor database/sql structs handling (#833) 2025-09-29 16:27:38 +02:00
Alejandro Ruiz 8e294f7a0b
Vai encoding benchmark (#742)
Also includes some misc changes.
2025-09-17 09:33:33 +02:00
Tom Lebreux a020084518
Fix slow labels queries (#762)
* Add PRAGMA temp_store=2 to write temporary tables in memory

* Only use DISTINCT when filtering labels

* Log long queries to debug

* Add unit test
2025-08-06 20:07:09 -04:00
Eric Promislow 5f08feeb5f
Reset db more selectively: approach #2 (#745)
* On Reset, remove all the sqlite files, not just the main one.

* Avoid resetting the database and cache when not needed.

Avoid resetting when a change doesn't bring in new schema fields.

Always reset when a CRD is deleted.

* Force a database reset when a new CRD is added.

* Improve code readability. No functional change.
2025-07-31 10:43:55 -07:00
Tom Lebreux 0c2c554c8c
Respect shouldEncrypt configuration when storing events (#725) 2025-07-15 09:57:28 -04:00
Tom Lebreux 127d37391d
Better gc (#717)
* More error wrapping for SQL cache

* Use context.Context instead of stopCh

* Move CacheFor to Info log level

* Implement time-based GC

* Set default GC param when running standalone steve
2025-07-11 12:48:19 -04:00
Eric Promislow 496a6f8968
Hard-wire external associations: 3 sections in, this one is 4/7 (#645)
* Continue rebasing.

* Wrote unit tests for external associations.

* Fix the generated SQL.

Some syntactic sugar (capitalizing the keywords), but use the 'ON' syntax on JOINs.

* whitespace fix post rebase

* We want "management.cattle.io.projects:spec.displayName" not "...spec.clusterName"

* Fix the database calls: drop the key

* Fix breakage during automatic rebase merging gone wrong.

* ws fix - NFC

* Post rebase-merge fixes

* Fix rebase-driven merge.

* Fix rebase breakage.

* go-uber v0.5.2 prefers real arg names to '<argN>'
2025-06-25 16:10:48 -07:00
Swastik Gour e17ca28461
Bumped dependencies to be compatible with k8s-v1.33 (#681)
Signed-off-by: swastik959 <Sswastik959@gmail.com>
2025-06-20 17:53:42 +05:30
Eric Promislow 2e8a0f2851
Support indexing on array-like fields (#673)
* Run tests using sqlite DB in a temp directory.

I was running into write-file errors which happens when two sqlite processes try to update the DB at the same time.

* Implement and test the extractBarredValue custom SQL function.

* Explain the DB path constants better.
2025-06-16 15:06:07 -07:00
Tom Lebreux b4db257cdb
Handle transaction failure due to canceled context.Context (#662)
* Re-order SQL event hooks so events are last

* Add QueryRowContext for single line queries

* Add test case for unknown resource version

* Properly check rows and close it

* More accurate error message when context.Context is canceled

* Re-order test check
2025-06-09 15:39:09 -04:00
Silvio Moioli 3350323f91
sql: propagate and use contexts (#465)
Previous SQLite-related code used context.Background() and context.TODO() because it was not developed with context awareness.

This commit propagates the main Steve context so that it can be used when interacting with SQL context-aware functions.

This PR removes all production-code use of context.Background() and context.TODO() and replaces test-code use of TODO with Background.

Contributes to rancher/rancher#47825
2025-02-12 09:46:10 +01:00
Silvio Moioli 772dc7577e
sql: use a closure to wrap transactions (#469)
This introduces the a `WithTransaction` function, which is then used for all transactional work in Steve.

Because `WithTransaction` takes care of all `Begin`s, `Commit`s and `Rollback`s, it eliminates the problem where forgotten open transactions can block all other operations (with long stalling and `SQLITE_BUSY` errors).

This also:

- merges together the disparate `DBClient` interfaces in one only `db.Client` interface with one unexported non-test implementation. I found this much easier to follow
- refactors the transaction package in order to make it as minimal as possible, and as close to the wrapped `sql.Tx` and `sql.Stmt` functions as possible, in order to reduce cognitive load when working with this part of the codebase
- simplifies tests accordingly
- adds a couple of known files to `.gitignore`
    
Credits to @tomleb for suggesting the approach: https://github.com/rancher/lasso/pull/121#pullrequestreview-2515872507
2025-02-05 10:05:52 +01:00
Tom Lebreux 9dd9b0f625
Move lasso SQL cache in Steve (#452)
* Copy pkg/cache/sql from lasso to pkg/sqlcache

* Rename import from github.com/rancher/lasso/pkg/cache/sql to github.com/rancher/steve/pkg/sqlcache

* Fix filter.Match -> filter.Matches

* go mod tidy

* Fix lint errors

* Remove lasso SQL cache mentions

* Fix more CI lint errors

* fix goimports

Signed-off-by: Silvio Moioli <silvio@moioli.net>

* fix tests (Match -> Matches)

Signed-off-by: Silvio Moioli <silvio@moioli.net>

* Fix Sort order

---------

Signed-off-by: Silvio Moioli <silvio@moioli.net>
Co-authored-by: Silvio Moioli <silvio@moioli.net>
2025-01-17 09:34:48 -05:00