Merge pull request #352 from hzxuzhonghu/develop

Add development guide
This commit is contained in:
Volcano Bot 2019-07-23 07:30:21 +05:30 committed by GitHub
commit c833efc373
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 699 additions and 0 deletions

76
code_of_conduct.md Normal file
View File

@ -0,0 +1,76 @@
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq

119
community-membership.md Normal file
View File

@ -0,0 +1,119 @@
# Volcano Community Membership
**Note :** This document keeps changing based on the status and feedback of Volcano Community.
This document gives a brief overview of the Volcano community roles with the requirements and responsibilities associated with them.
| Role | Requirements | Responsibilities | Privileges |
| -----| ---------------- | ------------ | -------|
| [Member](#member) | Sponsor from 2 approvers, active in community, contributed to Volcano | Welcome and guide new contributors | Volcano GitHub organization Member |
| [Approver](#approver) | Sponsor from 2 maintainers, has good experience and knowledge of domain, actively contributed to code and review | Review and approve contributions from community members | Write access to specific packages in relevant repository |
| [Maintainer](#maintainer) | Sponsor from 2 owners, shown good technical judgement in feature design/development and PR review | Participate in release planning and feature development/maintenance | Top level write access to relevant repository. Name entry in Maintainers file of the repository |
| [Owner](#owner) | Sponsor from 3 owners, helps drive the overall Volcano project | Drive the overall technical roadmap of the project and set priorities of activities in release planning | Volcano GitHub organization Admin access |
**Note :** It is mandatory for all Volcano community members to follow Volcano [Code of Conduct](./code_of_conduct.md).
## Member
Members are active participants in the community who contribute by authoring PRs,
reviewing issues/PRs or participate in community discussions on slack/mailing list.
### Requirements
- Sponsor from 2 approvers
- Enabled [two-factor authentication] on their GitHub account
- Actively contributed to the community. Contributions may include, but are not limited to:
- Authoring PRs
- Reviewing issues/PRs authored by other community members
- Participating in community discussions on slack/mailing list
- Participate in Volcano community meetings
### Responsibilities and privileges
- Member of the Volcano GitHub organization
- Can be assigned to issues and PRs and community members can also request their review
- Participate in assigned issues and PRs
- Welcome new contributors
- Guide new contributors to relevant docs/files
- Help/Motivate new members in contributing to Volcano
## Approver
Approvers are active members who have good experience and knowledge of the domain.
They have actively participated in the issue/PR reviews and have identified relevant issues during review.
### Requirements
- Sponsor from 2 maintainers
- Member for at least 2 months
- Have reviewed good number of PRs
- Have good codebase knowledge
### Responsibilities and Privileges
- Review code to maintain/improve code quality
- Acknowledge and work on review requests from community members
- May approve code contributions for acceptance related to relevant expertise
- Have 'write access' to specific packages inside a repo, enforced via bot
- Continue to contribute and guide other community members to contribute in Volcano project
## Maintainer
Maintainers are approvers who have shown good technical judgement in feature design/development in the past.
Has overall knowledge of the project and features in the project.
### Requirements
- Sponsor from 2 owners
- Approver for at least 2 months
- Nominated by a project owner
- Good technical judgement in feature design/development
### Responsibilities and privileges
- Participate in release planning
- Maintain project code quality
- Ensure API compatibility with forward/backward versions based on feature graduation criteria
- Analyze and propose new features/enhancements in Volcano project
- Demonstrate sound technical judgement
- Mentor contributors and approvers
- Have top level write access to relevant repository (able click Merge PR button when manual check-in is necessary)
- Name entry in Maintainers file of the repository
- Participate & Drive design/development of multiple features
## Owner
Owners are maintainers who have helped drive the overall project direction.
Has deep understanding of Volcano and related domain and facilitates major agreement in release planning
### Requirements
- Sponsor from 3 owners
- Maintainer for at least 2 months
- Nominated by a project owner
- Not opposed by any project owner
- Helped in driving the overall project
### Responsibilities and Privileges
- Make technical decisions for the overall project
- Drive the overall technical roadmap of the project
- Set priorities of activities in release planning
- Guide and mentor all other community members
- Ensure all community members are following Code of Conduct
- Although given admin access to all repositories, make sure all PRs are properly reviewed and merged
- May get admin access to relevant repository based on requirement
- Participate & Drive design/development of multiple features
**Note :** These roles are applicable only for Volcano github organization and repositories. Currently Volcano doesn't have a formal process for review and acceptance into these roles. We will come-up with a process soon.
[two-factor authentication]: https://help.github.com/articles/about-two-factor-authentication

142
contribute.md Normal file
View File

@ -0,0 +1,142 @@
# Welcome to Volcano!
- [Before you get started](#before-you-get-started)
- [Code of Conduct](#code-of-conduct)
- [Community Expectations](#community-expectations)
- [Getting started](#getting-started)
- [Your First Contribution](#your-first-contribution)
- [Find something to work on](#find-something-to-work-on)
- [Find a good first topic](#find-a-good-first-topic)
- [Work on an Issue](#work-on-an-issue)
- [File an Issue](#file-an-issue)
- [Contributor Workflow](#contributor-workflow)
- [Creating Pull Requests](#creating-pull-requests)
- [Code Review](#code-review)
- [Testing](#testing)
# Before you get started
## Code of Conduct
Please make sure to read and observe our [Code of Conduct](./code_of_conduct.md).
## Community Expectations
Volcano is a community project driven by its community which strives to promote a healthy, friendly and productive environment.
The goal of the community is to develop a volcano system which is useful for runnning high performance workloads such as AI,ML,Deep Learning Application on top of Kubernetes. To build a such volcano system at such scale requires the support of a community with similar aspirations.
- See [Community Membership](./community-membership.md) for a list of various community roles. With gradual contributions, one can move up in the chain.
# Getting started
- Read the [get started](docs/development/perepare-for-development.md) for developing code for Volcano
- Read the [setup](docs/development/development.md) for build/deploy instructions.
# Your First Contribution
We will help you to contribute in different areas like filing issues, developing features, fixing critical bugs and getting your work reviewed and merged.
If you have questions about the development process, feel free to jump into our [Slack Channel](https://github.com/volcano-sh/volcano/blob/master/slack-invitation) or join our [mailing list](https://groups.google.com/forum/#!forum/volcano-sh).
## Find something to work on
We are always in need of help, be it fixing documentation, reporting bugs or writing some code.
Look at places where you feel best coding practices aren't followed, code refactoring is needed or tests are missing.
Here is how you get started.
### Find a good first topic
There are [multiple repositories](https://github.com/volcano-sh/) within the Volcano organization.
Each repository has beginner-friendly issues that provide a good first issue.
For example, [Volcano-Issues](https://github.com/volcano-sh/volcano) has [help wanted](https://github.com/volcano-sh/volcano/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22) and [good first issue](https://github.com/volcano-sh/volcano/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22) labels for issues that should not need deep knowledge of the system.
We can help new contributors who wish to work on such issues.
Another good way to contribute is to find a documentation improvement, such as a missing/broken link.
#### Work on an issue
When you are willing to take on an issue, you can assign it to yourself. Just reply with `/assign` or `/assign @yourself` on an issue,
then the robot will assign the issue to you and your name will present at `Assignees` list.
### File an Issue
While we encourage everyone to contribute code, it is also appreciated when someone reports an issue.
Issues should be filed under the appropriate Volcano sub-repository.
*Example:* a Volcano issue should be opened to [Volcano](https://github.com/volcano-sh/volcano/issues).
Please follow the prompted submission guidelines while opening an issue.
# Contributor Workflow
Please do not ever hesitate to ask a question or send a pull request.
This is a rough outline of what a contributor's workflow looks like:
- Create a topic branch from where to base the contribution. This is usually master.
- Make commits of logical units.
- Make sure commit messages are in the proper format (see below).
- Push changes in a topic branch to a personal fork of the repository.
- Submit a pull request to [Volcano](https://github.com/volcano-sh/volcano).
- The PR must receive an approval from two maintainers.
## Creating Pull Requests
Pull requests are often called simply "PR".
Volcano generally follows the standard [github pull request](https://help.github.com/articles/about-pull-requests/) process.
In addition to the above process, a bot will begin applying structured labels to your PR.
The bot may also make some helpful suggestions for commands to run in your PR to facilitate review.
These `/command` options can be entered in comments to trigger auto-labeling and notifications.
Refer to its [command reference documentation](https://go.k8s.io/bot-commands).
## Code Review
To make it easier for your PR to receive reviews, consider the reviewers will need you to:
* follow [good coding guidelines](https://github.com/golang/go/wiki/CodeReviewComments).
* write [good commit messages](https://chris.beams.io/posts/git-commit/).
* break large changes into a logical series of smaller patches which individually make easily understandable changes, and in aggregate solve a broader issue.
* label PRs with appropriate reviewers: to do this read the messages the bot sends you to guide you through the PR process.
### Format of the commit message
We follow a rough convention for commit messages that is designed to answer two questions: what changed and why.
The subject line should feature the what and the body of the commit should describe the why.
```
scripts: add test codes for metamanager
this add some unit test codes to imporve code coverage for metamanager
Fixes #12
```
The format can be described more formally as follows:
```
<subsystem>: <what changed>
<BLANK LINE>
<why this change was made>
<BLANK LINE>
<footer>
```
The first line is the subject and should be no longer than 70 characters, the second line is always blank, and other lines should be wrapped at 80 characters. This allows the message to be easier to read on GitHub as well as in various git tools.
Note: if your pull request isn't getting enough attention, you can use the reach out on Slack to get help finding reviewers.
## Testing
There are multiple types of tests.
The location of the test code varies with type, as do the specifics of the environment needed to successfully run the test:
* Unit: These confirm that a particular function behaves as intended. Unit test source code can be found adjacent to the corresponding source code within a given package. These are easily run locally by any developer.
* Integration: These tests cover interactions of package components or interactions between Volcano components and Kubernetes control plane components like API server.
* End-to-end ("e2e"): These are broad tests of overall system behavior and coherence. The e2e tests are in [Volcano e2e](https://github.com/volcano-sh/volcano/tree/master/test/e2e).
Continuous integration will run these tests on PRs.

View File

@ -0,0 +1,163 @@
This document helps you get started using the Volcano code base.
If you follow this guide and find some problem, please take
a few minutes to update this file.
- [Building the code](#building-the-code)
- [Building docker images](#building-docker-images)
- [Building a specific docker image](#building-a-specific-docker-image)
- [Building the Volcano manifests](#building-the-volcano-manifests)
- [Cleaning outputs](#cleaning-outputs)
- [Running tests](#running-tests)
- [Auto-formatting source code](#auto-formatting-source-code)
- [Running the verification](#running-the-verification)
- [Adding dependencies](#adding-dependencies)
- [About testing](#about-testing)
## Cloning the code
You will need to clone the main `volcano` repo to `$GOPATH/src/volcano.sh/volcano` for
the below commands to work correctly.
## Building the code
To build volcano all components for your host architecture, go to
the source root and run:
```bash
make image_bins
```
the binaries will be generated at .../src/volcano.sh/volcano/_output/bin/linux/amd64/
but if we just make as below
```bash
make
```
then the binaries would be generated at .../src/volcano.sh/volcano/_output/bin/
To build a specific component for your host architecture, go to
the source root and run `make <component name>`:
```bash
make vc-scheduler
```
## Building docker images
Build the containers in your local docker cache:
```bash
make images
```
## Building a specific docker image
If you want to make a local change and test some component, say `vc-controllers`, you
could do:
Under volcano.sh/volcano repo
```bash
pwd
```
The path should be
```bash
.../src/volcano.sh/volcano
```
Set up environment variables HUB and TAG by
```bash
export HUB=docker.io/yourrepo
export TAG=citadel
```
Make some local change of the code, then build `vc-controllers`
```bash
make image.vc-controllers
```
## Building the Volcano manifests
Use the following command to build the deploy yaml files:
```bash
make generate-yaml
```
## Cleaning outputs
You can delete any build artifacts with:
```bash
make clean
```
## Running tests
### Running unit tests
You can run all the available unit tests with:
```bash
make unit-test
```
### Running e2e tests
You can run all the available e2e tests with:
```bash
make vcctl
make images
make e2e-test-kind
```
If you want to run e2e test in a existing cluster with volcano deployed, run the following:
```bash
export VK_BIN= need to set vcctl binary path (eg:.../src/volcano.sh/volcano/_output/bin/)
KUBECONFIG=${KUBECONFIG} go test ./test/e2e
```
## Auto-formatting source code
You can automatically format the source code to follow our conventions by going to the
top of the repo and entering:
```bash
./hack/update-gofmt.sh
```
## Running the verification
You can run all the verification we require on your local repo by going to the top of the repo and entering:
```bash
make verify
```
## Adding dependencies
Volcano uses [dep](https://github.com/golang/dep) to manage its dependencies.
If you want to add or update a dependency, running:
```bash
dep ensure -add dependency-name@version
```
## About testing
Before sending pull requests you should at least make sure your changes have
passed both unit and the verification. We only merge pull requests when
**all** tests are passing.
- Unit tests should be fully hermetic
- Only access resources in the test binary.
- All packages and any significant files require unit tests.
- Unit tests are written using the standard Go testing package.
- The preferred method of testing multiple scenarios or input is
[table driven testing](https://github.com/golang/go/wiki/TableDrivenTests)
- Concurrent unit test runs must pass.

View File

@ -0,0 +1,60 @@
This document helps you get started developing code for Volcano.
If you follow this guide and find some problem, please take
a few minutes to update this file.
Volcano components only have few external dependencies you
need to set up before being able to build and run the code.
- [Setting up Go](#setting-up-go)
- [Setting up Docker](#setting-up-docker)
- [Other dependencies](#other-dependencies)
- [Setting up Kubernetes](#setting-up-kubernetes)
- [Setting up personal access token](#setting-up-a-personal-access-token)
## Setting up Go
All Volcano components are written in the [Go](https://golang.org) programming language.
To build, you'll need a Go development environment. If you haven't set up a Go development
environment, please follow [these instructions](https://golang.org/doc/install)
to install the Go tools.
Volcano currently builds with Go 1.12
## Setting up Docker
Istio has a Docker build system for creating and publishing Docker images.
To leverage that you will need:
- **Docker platform:** To download and install Docker follow [these instructions](https://docs.docker.com/install/).
- **Docker Hub ID:** If you do not yet have a Docker ID account you can follow [these steps](https://docs.docker.com/docker-id/) to create one. This ID will be used in a later step when setting up the environment variables.
## Setting up Kubernetes
We require Kubernetes version 1.12 or higher with CRD support.
If you aren't sure which Kubernetes platform is right for you, see [Picking the Right Solution](https://kubernetes.io/docs/setup/).
* [Installing Kubernetes with Minikube](https://kubernetes.io/docs/setup/learning-environment/minikube/)
* [Installing Kubernetes with kops](https://kubernetes.io/docs/setup/production-environment/tools/kops/)
* [Installing Kubernetes with kubeadm-dind-cluster](https://github.com/kubernetes-sigs/kubeadm-dind-cluster)
### Setting up a personal access token
This is only necessary for core contributors in order to push changes to the main repos.
You can make pull requests without two-factor authentication
but the additional security is recommended for everyone.
To be part of the Volcano organization, we require two-factor authentication, and
you must setup a personal access token to enable push via HTTPS. Please follow
[these instructions](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/)
for how to create a token.
Alternatively you can [add your SSH keys](https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/).
### What's next?
Once you've set up the prerequisites, continue with [Using the Code Base](./development.md)
for more details about how to build & test Volcano.

View File

@ -0,0 +1,80 @@
# Feature Lifecycle
This document is to clarify definitions and differences between features and corresponding APIs
during different development stages (versions).
Each version has different level of stability, support time,
and requires different graduation criteria moving to next level:
* [Alpha](#alpha)
* [Beta](#beta)
* [GA](#ga)
## Alpha
The feature may be changed/upgraded in incompatible ways in the later versions.
The source code will be available in the release branch/tag as well as in the binaries.
Support for the feature can be stopped any time without notice.
The feature may have bugs.
The feature may also induce bugs in other APIs/Features if enabled.
The feature may not be completely implemented.
The API version names will be like v1alpha1, v1alpha2, etc. The suffixed number will be incremented by 1 in each upgrade.
### Graduation Criteria
- Each feature will start at alpha level.
- Should not break the functioning of other APIs/Features.
## Beta
The feature may not be changed/upgraded in incompatible ways in later versions,
but if changed in incompatible ways then upgrade strategy will be provided.
The source code will be available in the release branch/tag as well as in the binaries.
Support for the feature will not be stopped without 2 minor releases notice and will be present in at least next 2 minor releases.
The feature will have very less bugs.
The feature will not induce bugs in other APIs/Features if enabled.
The feature will be completely implemented.
The API version names will be like v1beta1, v1beta2, etc. The suffixed number will be incremented by 1 in each upgrade.
### Graduation Criteria
- Should have at least 50% coverage in e2e tests.
- Project agrees to support this feature for at least next 2 minor releases and notice of at least 2 minor releases will be given before stopping the support.
- Feature Owner should commit to ensure backward/forward compatibility in the later versions.
## GA
The feature will not be changed/upgraded in incompatible ways in the next couple of versions.
The source code will be available in the release branch/tag as well as in the binaries.
Support for the feature will not be stopped without 4 minor releases notice and will be present in at least next 4 minor releases.
The feature will not have major bugs as it will be tested completely as well as have e2e tests.
The feature will not induce bugs in other APIs/Features if enabled.
The feature will be completely implemented.
The API version names will be like v1, v2, etc.
### Graduation Criteria
- Should have complete e2e tests.
- Code is thoroughly tested and is reported to be very stable.
- Project will support this feature for at least next 4 minor releases and notice of at least 4 minor releases will be given before stopping support.
- Feature Owner should commit to ensure backward/forward compatibility in the later versions.
- Consensus from Volcano Maintainers as well as Feature/API Owners who use/interact with the Feature/API.

View File

@ -0,0 +1,22 @@
# Welcome to Volcano
Volcano is a batch system built on Kubernetes. It provides a suite of mechanisms currently missing from
Kubernetes that are commonly required by many classes of batch & elastic workload including:
1. machine learning/deep learning,
2. bioinformatics/genomics
3. other "big data" applications.
These types of applications typically run on generalized domain
frameworks like TensorFlow, Spark, PyTorch, MPI, etc, which Volcano integrates with.
### Why Vocano?
- // TODO better to add separate md file & Link
- Learn about Volcano [here](https://github.com/volcano-sh/volcano/blob/master/README.md)
### First Steps
To get the most out of Volcano, start by reviewing a few introductory topics:
- [perepare-for-development](../development/perepare-for-development.md) - preoaration for development
- [Setup](../development/development.md) - Install Volcano
- [Contributing](https://github.com/volcano-sh/volcano/blob/master/contribute.md) - Contribute to Volcano
- [Troubleshooting](../troubleshooting/troubleshooting.md) - Troubleshoot commonly occurring issues. GitHub issues are [here](https://github.com/volcano-sh/volcano/issues)

View File

@ -0,0 +1,17 @@
# Reporting bugs
If any part of the Volcano project has bugs or documentation mistakes, please let us know by opening an issue. We treat bugs and mistakes very seriously and believe no issue is too small. Before creating a bug report, please check that an issue reporting the same problem does not already exist.
To make the bug report accurate and easy to understand, please try to create bug reports that are:
- Specific. Include as much details as possible: which version, what environment, what configuration, etc. If the bug is related to running the volcano component, please attach the component log (the starting log with component configuration is especially important).
- Reproducible. Include the steps to reproduce the problem. We understand some issues might be hard to reproduce, please includes the steps that might lead to the problem.
- Isolated. Please try to isolate and reproduce the bug with minimum dependencies. It would significantly slow down the speed to fix a bug if too many dependencies are involved in a bug report.
- Unique. Do not duplicate existing bug report.
- Scoped. One bug per report. Do not follow up with another bug inside one report.
We might ask for further information to locate a bug. A duplicated bug report will be closed.

View File

@ -0,0 +1,14 @@
# Support
If you need support, start with the [troubleshooting guide](../troubleshooting/troubleshooting.md), and work your way through the process that we've outlined.
## Community
**Slack channel:**
We use Slack for public discussions. To chat with us or the rest of the community, join us in the [Volcano Slack](https://volcano-sh.slack.com) team channel #general. To sign up, use our Slack inviter link [here](https://join.slack.com/t/volcano-sh/shared_invite/enQtNTU5NTU3NDU0MTc4LTgzZTQ2MzViNTFmNDg1ZGUyMzcwNjgxZGQ1ZDdhOGE3Mzg1Y2NkZjk1MDJlZTZhZWU5MDg2MWJhMzI3Mjg3ZTk).
**Mailing List**
Please sign up on our [mailing list](https://groups.google.com/forum/#!forum/volcano-sh)

View File

@ -0,0 +1,6 @@
# FAQs
This page contains a few commonly occuring questions.
For further support please contact us using the [support page](../getting-started/support.md)
// TODO need to update