A new function, `isTypedArray` was added to the `platform` interface by #7181
and first published in tfjs-core 4.2.0. This made 4.2.0 incompatible with
earlier versions of backends that implemented `platform`, such as node and
react-native. This change adds a fallback to the use of `isTypedArray` so
earlier versions of platforms that don't implement `isTypedArray` will not throw
an error.
Note that the fallback behavior may not be perfect, such as when running Jest
tests in node. See #7175 for more details and upgrade all @tensorflow scoped
packages to ^4.2.0 to avoid this.
This major release upgrades TypeScript from 3.5.3 to 4.8.4 (https://github.com/tensorflow/tfjs/pull/6346). It is intended to be as non-breaking as possible, but it technically has breaking changes for users of `typescript<4.4`. All other users should be unaffected.
When compiling projects that depend on TFJS with `typescript<4.4`, the following error will appear.
```
node_modules/@webgpu/types/dist/index.d.ts:587:16 - error TS2304: Cannot find name 'PredefinedColorSpace'.
587 colorSpace?: PredefinedColorSpace;
~~~~~~~~~~~~~~~~~~~~
...
```
This can be fixed by upgrading TypeScript to 4.4.2 or greater, or by adding the file `predefined_color_space.d.ts` (name and path can be changed) with the following contents to your project to define the missing type. If this file is added, it will need to be removed when TypeScript is upgraded past 4.3.
```typescript
type PredefinedColorSpace = "display-p3" | "srgb";
```
`typescript<3.7` has the following additional error.
```
node_modules/@tensorflow/tfjs-core/dist/engine.d.ts:127:9 - error TS1086: An accessor cannot be declared in an ambient context.
127 get backend(): KernelBackend;
~~~~~~~
...
```
Enabling [`skipLibCheck`](https://www.typescriptlang.org/tsconfig#skipLibCheck) suppresses this error, and upgrading to at least TypeScript 3.6.2 fixes it (although the above fix for `PredefinedColorSpace` will also need to be applied).
Currently, we run Verdaccio tests only when we do a release. Since they're only tested after choosing a release commit from nightly CI, they are often a point of failure. This PR makes them run during nightly tests, so we can identify issues with them before creating a release PR.
This PR also enables Bazel sandboxing, which fixes the rules_nodejs linker issue.
Make build-deps-for check for Bazel dependencies transitively instead of just on the packages passed to it. Additionally, change the behavior of `build-deps-for` so that when it is passed a Bazel package, it will directly build the package instead of building just its dependencies.
Co-authored-by: Linchenn <40653845+Linchenn@users.noreply.github.com>
Improve link_package speed by only building the dependencies that are required and by building them all in a single bazel build command.
Remove the link_package_core package, which is no longer needed now that link_package can build just the required dependencies.
Add the "repository" field to packages that were missing it and update the field for packages that were migrated into the monorepo. See #5770 for details on why this is being done.
This PR builds tfjs-core and tfjs-backend-cpu using Bazel and makes the build outputs available to the other packages in the monorepo. It also sets up Bazel rules and macros that facilitate converting other packages in the monorepo to Bazel. For more details, see #5133.
* [tfjs-node] Make `yarn build` works on Windows
FEATURE
* Add `yarn` before node-pre-gyp command
This made command node-pre-gyp works on Windows
* Align tfjs-node-gpu package.json with tfjs-node
* Run ts-node tests with --transpile-only to increase speed
* Run 'yarn build' before testing with ts-node
* [cpu] Move run_tests.ts to /src so it is typechecked
* [union] Run test-tools with typechecking
* Remove old tfjs-node integration tests
* Add test-dev to packages that use ts-node
* [core] Remove dependency on union that was added earlier in this PR
* Add a command to each package
* Remove --transpile-only flag from targets that never run through tsc
* Run CI on a single machine by generating cloudbuild files
Instead of running each CI test on a separate Google Cloud machine, generate a cloudbuild.yml file to test all affected packages on a single machine. The benifits of this strategy include the following:
* Fewer machines are required for a single test (2 instead of up to 12), so queuing should happen much less often.
* Packages only need to be built once, instead of once per machine.
* Multithreaded steps have access to more cores.
When CI is run, the root cloudbuild.yml is launched on Google Cloud. This file has only three steps: Install yarn dependencies, run `yarn generate-cloudbuild-for-packages`, and run `run-build.sh`. The second step finds what packages were affected by the change and combines their cloudbuild.yml files into a single `cloudbuild_generated.yml` file, and the third step launches the generated build on Google Cloud. The steps in the generated build are set to `waitFor` each other according to the dependency tree `scripts/package_dependencies.json`, and each package's `build-deps` step is removed in favor of waiting for its dependencies to be built.
Each package's cloudbuild file still works as expected, and there are two new yarn commands at the root:
* `yarn test-packages` tests all packages affected by a change. It performs the same steps as CI.
* `yarn generate-cloudbuild-for-packages [PACKAGE]...` generates the cloudbuild file to test each `PACKAGE` and packages it might affect. If given no packages, it detects what packages have been changed in the current PR.
* Verify the dependency graph is a valid graph
* Add dependencies to e2e
* Use the node:12 docker image for tfjs-node CI
tfjs-node builds N-API bindings that end-to-end tests need to use. End-to-end uses our custom docker image (gcr.io/learnjs-174218/release) since it has to build tfjs-backend-wasm when it's run by itself (which requires emscripten). Our custom image uses node 12, and is incompatible with node 10 N-API bindings. Eventually, we'll want to move everything that's using node:10 to node:12, but that's out of the scope of this commit.
* Remove TODO and allow changes to generate_cloudbuild.js to trigger a full rebuild
* Add tfjs-backend-cpu to tfjs-data in package_dependencies.json
tfjs-backend-cpu is listed in tfjs-data's package.json as a linked dependency, but was missing form package_dependencies.json.
* Don't have tfjs-backend-webgpu build its dependencies in CI
Move tfjs-backend-webgpu's dependency building to the 'build-deps' step so it can be removed during CI.
This commit also updates the test cloudbuild files that should have been updated in f9c1fdd814. It would probably be best to amend that commit, but last time I tried force-pushing in GitHub, I broke the PR I was working on.
* Revert changes in tfjs-backend-webgpu to master
tfjs-backend-webgpu is now pinned at v2.7.0 for all its tfjs dependencies.
* Add the update-cloudbuild-tests script to package.json
update-cloudbuild-tests updates the change detection tests for scripts/generate_cloudbuild.js. It should be run iff a change affects cloudbuild files or the dependency tree.
* Make tfjs-backend-webgpu tests wait for yarn to run
* Add a destination argument to generate_cloudbuild_for_packages.js
Co-authored-by: Ping Yu <4018+pyu10055@users.noreply.github.com>
FEATURE
INTERNAL
* Run CI on a single machine by generating cloudbuild files.
Instead of running each CI test on a separate Google Cloud machine, generate a cloudbuild.yml file to test all affected packages on a single machine. The benifits of this strategy include the following:
* Fewer machines are required for a single test (2 instead of up to 12), so less queueing is required.
* Packages only need to be built once, instead of once per machine.
* Multithreaded steps have access to more cores.
When CI is run, the root cloudbuild.yml is launched on Google Cloud. This file has only two steps: Install yarn dependencies and run `yarn test-affected-packages`. The latter step finds what packages were affected by the change and combines their cloudbuild.yml files into a single `cloudbuild_generated.yml` file. The steps in this file are set to `waitFor` each other according to the dependency tree `scripts/package_dependencies.json`, and each package's `build-deps` step is removed in favor of waiting for its dependencies to be built.
Each package's cloudbuild file still works as expected, and there are a few new yarn commands at the root:
`yarn test-packages` tests all packages affected by a change. It's run during CI.
`yarn test-packages [PACKAGE]...` tests a given set of packages, including all packages they might affect if changed.
* Split generating the cloudbuild from running it
Google Cloud uses a separate image for node and for spawning more gcloud instances. The node image does not have the gcloud command, so we have to generate the cloudbuild file in one step and launch gcloud in another step.
Also fix the `rm` command in find_packages_with_diff.js to remove recursively.
* Add `yarn` to the `waitFor` field of webgpu tests
* Modularize Sum and Softmax. (#4148)
DEV
* Lint files
* Fix package dependencies for inference and vis
* Run CI on a single machine by generating cloudbuild files.
Instead of running each CI test on a separate Google Cloud machine, generate a cloudbuild.yml file to test all affected packages on a single machine. The benifits of this strategy include the following:
* Fewer machines are required for a single test (2 instead of up to 12), so less queueing is required.
* Packages only need to be built once, instead of once per machine.
* Multithreaded steps have access to more cores.
When CI is run, the root cloudbuild.yml is launched on Google Cloud. This file has only two steps: Install yarn dependencies and run `yarn test-affected-packages`. The latter step finds what packages were affected by the change and combines their cloudbuild.yml files into a single `cloudbuild_generated.yml` file. The steps in this file are set to `waitFor` each other according to the dependency tree `scripts/package_dependencies.json`, and each package's `build-deps` step is removed in favor of waiting for its dependencies to be built.
Each package's cloudbuild file still works as expected, and there are a few new yarn commands at the root:
`yarn test-packages` tests all packages affected by a change. It's run during CI.
`yarn test-packages [PACKAGE]...` tests a given set of packages, including all packages they might affect if changed.
* Split generating the cloudbuild from running it
Google Cloud uses a separate image for node and for spawning more gcloud instances. The node image does not have the gcloud command, so we have to generate the cloudbuild file in one step and launch gcloud in another step.
Also fix the `rm` command in find_packages_with_diff.js to remove recursively.
* Add `yarn` to the `waitFor` field of webgpu tests
* Lint files
* Fix package dependencies for inference and vis
* modularize node kernels (#4154)
Modularizes onesLike, zerosLike, unpack/unstack, batchMatMul, slice, isFinite, isInf, IsNan, reciprocal, squaredDifference
DEV
* Enable incremental typescript builds (#4092)
FEATURE
INTERNAL
Enable incremental builds in the root tsconfig.json to increase build speed during development. For example, running `yarn build` in tfjs-backend-cpu takes ~43 seconds without incremental builds and ~30 seconds with incremental builds (after the first build).
Co-authored-by: Ping Yu <4018+pyu10055@users.noreply.github.com>
* Fix typo in .github/stale.yml (#3642)
INTERNAL
* Address some of the comments
* Re-lint everything with VSCode.
* Hoist consts and use UPPER_SNAKE_CASE.
* Require each step to have an id and add ids to all steps in converter and vis.
* Use argparse for 'generate_cloudbuild_for_packages.js'
* Remove inference, vis, and react-native dependencies in package_dependencies.json
tfjs-inference, vis, and react-native use pinned versions of other tfjs packages, so they do not need to be tested when those pinned packages change.
* Fix generate_cloudbuild_for_packages
generate_cloudbuild_for_packages.js was using argparse functions from a later version of argparse that are not available in 1.0.10.
* [webgl] Modularize batchMatMul, sum. (#4140)
FEATURE
* Add Prod kernel to WASM backend. (#4138)
FEATURE
Co-authored-by: Ann Yuan <annyuan@gmail.com>
* Write tests for generate_cloudbuild.js
Write tests to detect if generate_cloudbuild's output changes. These tests are not unit tests, but they alert us if a change unexpectedly changes the project structure or the generation of cloudbuild files.
* Run CI on a single machine by generating cloudbuild files.
Instead of running each CI test on a separate Google Cloud machine, generate a cloudbuild.yml file to test all affected packages on a single machine. The benifits of this strategy include the following:
* Fewer machines are required for a single test (2 instead of up to 12), so less queueing is required.
* Packages only need to be built once, instead of once per machine.
* Multithreaded steps have access to more cores.
When CI is run, the root cloudbuild.yml is launched on Google Cloud. This file has only two steps: Install yarn dependencies and run `yarn test-affected-packages`. The latter step finds what packages were affected by the change and combines their cloudbuild.yml files into a single `cloudbuild_generated.yml` file. The steps in this file are set to `waitFor` each other according to the dependency tree `scripts/package_dependencies.json`, and each package's `build-deps` step is removed in favor of waiting for its dependencies to be built.
Each package's cloudbuild file still works as expected, and there are a few new yarn commands at the root:
`yarn test-packages` tests all packages affected by a change. It's run during CI.
`yarn test-packages [PACKAGE]...` tests a given set of packages, including all packages they might affect if changed.
* Split generating the cloudbuild from running it
Google Cloud uses a separate image for node and for spawning more gcloud instances. The node image does not have the gcloud command, so we have to generate the cloudbuild file in one step and launch gcloud in another step.
Also fix the `rm` command in find_packages_with_diff.js to remove recursively.
* Add `yarn` to the `waitFor` field of webgpu tests
* Lint files
* Fix package dependencies for inference and vis
* Address some of the comments
* Re-lint everything with VSCode.
* Hoist consts and use UPPER_SNAKE_CASE.
* Require each step to have an id and add ids to all steps in converter and vis.
* Use argparse for 'generate_cloudbuild_for_packages.js'
* Remove inference, vis, and react-native dependencies in package_dependencies.json
tfjs-inference, vis, and react-native use pinned versions of other tfjs packages, so they do not need to be tested when those pinned packages change.
* Fix generate_cloudbuild_for_packages
generate_cloudbuild_for_packages.js was using argparse functions from a later version of argparse that are not available in 1.0.10.
* Write tests for generate_cloudbuild.js
Write tests to detect if generate_cloudbuild's output changes. These tests are not unit tests, but they alert us if a change unexpectedly changes the project structure or the generation of cloudbuild files.
Co-authored-by: Na Li <linazhao@google.com>
Co-authored-by: Yannick Assogba <yassogba@google.com>
Co-authored-by: Ping Yu <4018+pyu10055@users.noreply.github.com>
Co-authored-by: Rajeshwar Reddy T <43972606+rthadur@users.noreply.github.com>
Co-authored-by: Ann Yuan <annyuan@gmail.com>
Co-authored-by: miaowzhang <miaowzhang@gmail.com>
gaikwadrahul8