# KFP v2 API Test Framework

This directory contains the comprehensive integration test suite for Kubeflow Pipelines (KFP) v2 APIs. The test framework is built using **Ginkgo** and **Gomega** testing libraries and provides end-to-end testing for all API services.

## Table of Contents

1. [Test Framework Architecture](#test-framework-architecture)
2. [Test Coverage](#test-coverage)
3. [Test Creation Process](#test-creation-process)
4. [Directory Structure](#directory-structure)
5. [Running Tests](#running-tests)
6. [Contributing](#contributing)

## Test Framework Architecture

### Ginkgo & Gomega Integration

The test framework leverages:

- **Ginkgo v2**: BDD-style testing framework for organizing and running tests
- **Gomega**: Matcher library for assertions and expectations

#### Key Framework Components

**1. Test Suite Setup (`integration_suite_test.go`)**
- Global `BeforeSuite()`: Initializes API clients, creates directories, sets up Kubernetes client
- Global `BeforeEach()`: Sets up test-specific variables and common test data
- Global `AfterEach()`: Performs cleanup operations (deletes pipelines, runs, experiments)
- `ReportAfterEach()`: Captures logs and test reports on failure

**2. Configuration & Flags (`flags.go`)**
- Configurable test parameters via command-line flags
- Environment-specific settings (namespace, timeouts, debug mode)
- Test execution modes (dev mode, integration tests, proxy tests)

```go
// Example flag usage
var namespace = flag.String("namespace", "kubeflow", "The namespace ml pipeline deployed to")
var isDebugMode = flag.Bool("isDebugMode", false, "Whether to enable debug mode")
```

### Logging System (`logger/`)

**Custom Logger Implementation:**
- Integrates with Ginkgo's `GinkgoWriter` for test output
- Provides structured logging with format support
- Captures logs for failed test analysis

```go
// Usage example
logger.Log("Uploading pipeline file %s", pipelineFile)
logger.Log("Created Pipeline Run with id: %s", runID)
```

### Utility Classes (`utils/`)

**Core Utility Files:**
- `config_utils.go`: Client configuration and connection setup
- `pipeline_utils.go`: Pipeline CRUD operations and validation
- `kubernetes_utils.go`: Kubernetes cluster interactions
- `file_utils.go`: File system operations for test data

**Example Utility Usage:**
```go
// Pipeline operations
uploadedPipeline := utils.CreatePipeline(client, pipelineFile)
utils.DeletePipeline(client, pipelineID)

// Run operations
utils.WaitForRunToBeInState(runClient, runID, []State{RUNNING})
utils.TerminatePipelineRun(runClient, runID)
```

### Custom Matchers (`matcher/`)

**Specialized Assertion Matchers:**
- `custom_matcher.go`: Generic map comparison utilities
- `pipeline_matcher.go`: Pipeline-specific validation matchers

```go
// Custom matcher usage
MatchMaps(actualMap, expectedMap, "pipeline parameters")
```

### Constants & Configuration (`constants/`)

**Static Test Labels:**
- `test_type.go`: Test categories (Smoke, CriticalOnly, FullRegression)

**Example Test Labeling:**
```go
var _ = Describe("Pipeline API Tests", Label(FullRegression), func() {
    // Test implementation
})
```
**Open Text Labels:**

Tests can also be labeled with any text, and these labels can further be used to filter out during execution
**Example Labeling:**
```go
var _ = Describe("Pipeline API Tests", Label("Positive", "Pipeline"), func() {
    // Test implementation
})
```

**Current List of Labels**
```
Smoke, CriticalOnly, FullRegression, "Positive", "Negative", "PipelineUpload", "ApiServerTests"
```

### Reports Generation

**Automated Report Generation:**
- **JUnit XML Reports**: `reports/api.xml` for CI/CD integration
- **JSON Reports**: `reports/api.json` for detailed analysis
- **Test Logs**: Individual test logs in `logs/` directory
- **Pod Logs**: Captured on test failures for debugging

```go
// Report configuration
reporterConfig.JUnitReport = filepath.Join(testReportDirectory, junitReportFilename)
reporterConfig.JSONReport = filepath.Join(testReportDirectory, jsonReportFilename)
```

## Test Coverage

### API Service Test Distribution

Each API service has dedicated test files with comprehensive endpoint coverage:

| API Service             | Test File                            | Primary Focus                        | 
|-------------------------|--------------------------------------|--------------------------------------|
| **Pipeline Upload API** | `pipeline_upload_api_test.go`        | File upload, validation, versioning  |

### Endpoint Coverage Details

**Pipeline API Coverage:**
- ✅ List operations (pagination, sorting, filtering)
- ✅ CRUD operations (create, read, update, delete)
- ✅ Version management
- ✅ Namespace isolation
- ✅ Error handling and validation

**Pipeline Run API Coverage:**
- ✅ Run creation and execution
- ✅ State transitions (PENDING → RUNNING → SUCCESS/FAILED)
- ✅ Run termination and cancellation
- ✅ Archive/unarchive operations
- ✅ Experiment association
- ✅ Parameter handling and validation

**Experiment API Coverage:**
- ✅ Experiment lifecycle management
- ✅ Run association and disassociation
- ✅ Archive/unarchive workflows
- ✅ Filtering and search operations

### Test Categorization

**By Test Type:**
- **Smoke Tests**: Critical path validation
- **Regression Tests**: Comprehensive feature coverage
- **Integration Tests**: Cross-service interaction validation

## Test Creation Process

### 1. Test Structure Pattern

All tests follow a consistent BDD structure using Ginkgo's descriptive syntax:

```go
var _ = Describe("API Feature Description >", Label("Positive", "ServiceName", FullRegression), func() {
    Context("Specific Scenario Group >", func() {
        It("Should perform specific action", func() {
            // Test implementation
        })
    })
})
```

### 2. Test Implementation Lifecycle

**Setup Phase:**
```go
BeforeEach(func() {
    logger.Log("################### Setup before each test #####################")
    // Initialize test-specific variables
    // Create test data
    // Setup API clients
})
```

**Test Execution:**
```go
It("Should validate specific behavior", func() {
    // 1. Arrange: Setup test data
    logger.Log("Starting test: %s", testDescription)
    
    // 2. Act: Execute the operation
    result := performAPICall(testData)
    
    // 3. Assert: Validate results
    Expect(result).To(Equal(expectedResult))
    Expect(result.Status).To(Equal("SUCCESS"))
})
```

**Cleanup Phase:**
```go
AfterEach(func() {
    logger.Log("################### Global Cleanup after each test #####################")
    // Clean up created resources
    // Delete test data
    // Reset state
})
```

### 3. Logging Best Practices

**Structured Logging:**
```go
// Test phase logging
logger.Log("################### Setup Phase #####################")
logger.Log("Creating pipeline with name: %s", pipelineName)
logger.Log("Pipeline created successfully with ID: %s", pipelineID)
logger.Log("################### Cleanup Phase #####################")
```

**Error Context Logging:**
```go
logger.Log("Test failed... Capturing pod logs")
podLogs := utils.ReadContainerLogs(k8Client, namespace, "pipeline-api-server", &testStartTimeUTC)
AddReportEntry("Pod Log", podLogs)
```

### 4. Resource Management

**Automatic Cleanup:**
- All created resources are tracked in global arrays
- Cleanup happens in `AfterEach()` regardless of test outcome
- Resources include: pipelines, pipeline runs, experiments

```go
// Resource tracking
createdPipelines = []*upload_model.V2beta1Pipeline{}
createdRunIds = make([]string, 0)
createdExperimentIds = make([]string, 0)

// Automatic cleanup
for _, pipeline := range createdPipelines {
    utils.DeletePipeline(pipelineClient, pipeline.PipelineID)
}
```

### 5. Error Handling & Validation

**Assertion Patterns:**
```go
// Standard assertions
Expect(err).NotTo(HaveOccurred(), "Operation should succeed")
Expect(response.Status).To(Equal(expectedStatus))
Expect(len(results)).To(BeNumerically(">=", 1))

// Custom matcher usage
MatchMaps(actualParameters, expectedParameters, "pipeline parameters")
```

## Directory Structure

```
backend/test/v2/api/
├── README.md                          # This documentation
├── integration_suite_test.go          # Main test suite setup
├── flags.go                          # Test configuration flags
├── constants/                        # Test constants and enums
│   └── test_type.go                  # Test type categories
├── logger/                           # Logging utilities
│   └── logger.go                     # Custom logger implementation
├── matcher/                          # Custom assertion matchers
│   ├── custom_matcher.go             # Generic map matchers
│   └── pipeline_matcher.go           # Pipeline-specific matchers
├── utils/                            # Utility functions
│   ├── config_utils.go               # Client configuration
│   ├── file_utils.go                 # File system operations
│   ├── pipeline_utils.go             # Pipeline operations
│   ├── pipeline_version_utils.go     # Version management
├── logs/                             # Test execution logs
├── reports/                          # Generated test reports
│   ├── api.xml                       # JUnit XML report
│   └── api.json                      # JSON test report
├── *_api_test.go                     # Individual API test files
├── pipeline_e2e_test.go              # End-to-end tests
└── argo_workflow_converter.go        # Workflow conversion utilities
```

## Running Tests

### Basic Test Execution

```bash
# Run all API tests
go test -v ./backend/test/v2/api/

# Run with specific flags
go test -v ./backend/test/v2/api/ \
  -namespace=kubeflow \
  -isDebugMode=true \
  -runIntegrationTests=true
```

### Test Filtering

```bash
# Run only smoke tests
ginkgo -v --label-filter="Smoke" ./backend/test/v2/api/

# Run pipeline-specific tests
ginkgo -v --label-filter="Pipeline" ./backend/test/v2/api/
```

### Configuration Options

| Flag                  | Default    | Description                   |
|-----------------------|------------|-------------------------------|
| `namespace`           | `kubeflow` | Target Kubernetes namespace   |
| `isDebugMode`         | `false`    | Enable detailed debug logging |
| `runIntegrationTests` | `true`     | Execute integration tests     |
| `isKubeflowMode`      | `false`    | Run in full Kubeflow mode     |
| `cacheEnabled`        | `true`     | Enable pipeline caching       |
| `podLogLimit`         | `50000000` | Limit for captured pod logs   |

## Contributing

### Adding New Tests

1. **Create test file**: Follow naming convention `*_api_test.go`
2. **Import required packages**: Include framework imports and utilities
3. **Define test structure**: Use BDD patterns with proper labeling
4. **Implement test logic**: Follow the setup → act → assert → cleanup pattern
5. **Add proper logging**: Use structured logging throughout
6. **Update documentation**: Document new test coverage

### Test Development Guidelines

- **Use descriptive test names** that clearly indicate the test purpose
- **Implement proper cleanup** to avoid test pollution
- **Add appropriate labels** for test categorization and filtering
- **Include comprehensive logging** for debugging failed tests
- **Follow consistent code patterns** established in existing tests
- **Validate all assumptions** with explicit assertions

### Code Review Checklist

- [ ] Tests follow BDD structure and naming conventions
- [ ] Proper resource cleanup is implemented
- [ ] Comprehensive logging is included
- [ ] Test labels are correctly applied
- [ ] Error cases are handled appropriately
- [ ] Documentation is updated if needed 