COMET — CANFAR CI/CD Toolkit

v1.7.9

Harbor host: images.canfar.net, Arc host: ws-uv.canfar.net, port: 64022, status:

1. Login · 2. Repository · 3. CANFAR · 4. Deploy · 5. Finish

Sign in to a repository provider

GitHub

Not connected.

GitLab

Not connected.

Sign in to access CANFAR

Repository source

Choose template

Templates are curated starting points. Pick one, then use Apply Template to add selected pieces before COMET commits anything. Choose None for a deployment-only baseline, use CanDIAmazing for a fuller Python starter with docs, tests, and CI/CD sourced from github.com/CanDIAPL/candiamazing, or choose Custom... to prepare one of your own template repositories. AI Fit is language-agnostic, so it can adapt and translate relevant template pieces for non-Python repos too.

Template

3/day

Create a new repository

Use an existing repository

CANFAR

Selected repository: (none)

CADC Username

Arc Host

Port

CANFAR Remote Directory

Deploy into nested repo folder ? When enabled, COMET deploys into a subfolder named after your repository inside the directory you selected. For example, selecting /projects/my-project will deploy into /projects/my-project/<repo>.

Secrets COMET will add to your repository:

What COMET commits for you

Below is what COMET will commit for your current selection.

Commit Message

File Overview

Sort by

0 Existing

0 Updated

0 New

0 Collisions

.github/workflows/deploy-canfar.yml Filename collision: this file already exists in the selected repo/branch. Rename one of the files or the existing file will be overwritten. New file COMET will add to the repository. ? GitHub Actions workflow that deploys your repo to CANFAR (or builds/pushes a container if you choose Container mode). Keep this if you want automatic deploys on every push.

name: Deploy to CANFAR

on:
  push:
    branches: [ main ]
  workflow_dispatch:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.x'

      - name: Install test dependencies
        run: |
          python -m pip install --upgrade pip pytest pytest-cov
          if [ -f requirements.txt ]; then
            python -m pip install -r requirements.txt || true
          fi
          python -m pip install numpy
          if [ -f pyproject.toml ] || [ -f setup.py ] || [ -f setup.cfg ]; then
            python -m pip install -e .
          fi

      - name: Generate test fixtures (optional)
        env:
          COMET_TEST_DATA_DIR: tests/data
          COMET_TEST_DATA_SOURCE_DIR: ${{ secrets.COMET_TEST_DATA_SOURCE_DIR }}
        run: |
          set -euo pipefail
          if [ -f tests/_comet_generate_fixtures.py ]; then
            python tests/_comet_generate_fixtures.py
          else
            echo "tests/_comet_generate_fixtures.py not found; skipping"
          fi

      - name: Run pytest
        env:
          COMET_TEST_DATA_DIR: tests/data
          COMET_TEST_DATA_SOURCE_DIR: ${{ secrets.COMET_TEST_DATA_SOURCE_DIR }}
        run: |
          set -euo pipefail
          if [ -d tests ]; then
            COV_ARGS=""
            if [ -d src ]; then
              COV_ARGS="--cov=src"
            else
              for candidate in *.py */__init__.py; do
                [ -e "$candidate" ] || continue
                case "$candidate" in tests/*|test/*|setup.py) continue ;; esac
                if [ -f "$candidate" ]; then
                  module="${candidate%.py}"
                else
                  module="${candidate%/__init__.py}"
                fi
                [ -n "$module" ] || continue
                COV_ARGS="$COV_ARGS --cov=$module"
              done
            fi
            if [ -z "$COV_ARGS" ]; then
              COV_ARGS="--cov=. --cov-omit=tests/*,test/*"
            fi
            pytest tests -q $COV_ARGS --cov-report=term-missing --cov-fail-under=75
          else
            echo "tests/ not found; skipping"
          fi

  prepare:
    needs: tests
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Validate required inputs
        env:
          CANFAR_USER: ${{ secrets.CANFAR_USER }}
          CANFAR_SSH_PRIVATE_KEY: ${{ secrets.CANFAR_SSH_PRIVATE_KEY }}
          CANFAR_REMOTE_DIR: ${{ secrets.CANFAR_REMOTE_DIR }}
          CANFAR_SSH_HOST: ${{ secrets.CANFAR_SSH_HOST }}
          CANFAR_SSH_PORT: ${{ secrets.CANFAR_SSH_PORT }}
        run: |
          set -euo pipefail
          missing=0
          if [ -z "${CANFAR_USER:-}" ]; then
            echo "ERROR: Missing required secret CANFAR_USER" >&2
            missing=1
          fi
          if [ -z "${CANFAR_SSH_PRIVATE_KEY:-}" ]; then
            echo "ERROR: Missing required secret CANFAR_SSH_PRIVATE_KEY" >&2
            missing=1
          fi
          if [ -z "${CANFAR_REMOTE_DIR:-}" ]; then
            echo "ERROR: Missing required secret CANFAR_REMOTE_DIR (e.g., /projects/CanDIAPL/HiPS)" >&2
            missing=1
          fi
          if [ -z "${CANFAR_SSH_HOST:-}" ]; then
            echo "ERROR: Missing required secret CANFAR_SSH_HOST (SSH-accessible CANFAR host)" >&2
            missing=1
          fi
          if [ -z "${CANFAR_SSH_PORT:-}" ]; then
            echo "Using default CANFAR_SSH_PORT=64022"
            echo "CANFAR_SSH_PORT=64022" >> $GITHUB_ENV
          else
            echo "CANFAR_SSH_PORT=${CANFAR_SSH_PORT}" >> $GITHUB_ENV
          fi
          if [ "$missing" -ne 0 ]; then
            exit 1
          fi

      - name: Stage files for upload (exclude .git)
        run: |
          set -euo pipefail
          rm -rf /tmp/canfar_sync
          mkdir -p /tmp/canfar_sync
          rsync -az --delete --exclude '.git' ./ /tmp/canfar_sync/

      - name: Archive staged files
        run: |
          set -euo pipefail
          cd /tmp
          tar czf "$GITHUB_WORKSPACE/canfar_sync.tgz" canfar_sync

      - name: Upload staged artifact
        uses: actions/upload-artifact@v4
        with:
          name: canfar-sync-src
          path: canfar_sync.tgz

  sync:
    needs: prepare
    runs-on: ubuntu-latest
    steps:
      - name: Download staged files
        uses: actions/download-artifact@v4
        with:
          name: canfar-sync-src

      - name: Extract staged files
        run: |
          set -euo pipefail
          tar xzf canfar_sync.tgz

      - name: Install SSH key
        env:
          CANFAR_SSH_PRIVATE_KEY: ${{ secrets.CANFAR_SSH_PRIVATE_KEY }}
        run: |
          set -euo pipefail
          mkdir -p ~/.ssh
          cat <<'EOF' > ~/.ssh/id_canfar
          ${{ secrets.CANFAR_SSH_PRIVATE_KEY }}
          EOF
          chmod 600 ~/.ssh/id_canfar
          eval "$(ssh-agent -s)"
          ssh-add ~/.ssh/id_canfar

      - name: Configure SSH known_hosts (ignore scan failures)
        env:
          CANFAR_SSH_HOST: ${{ secrets.CANFAR_SSH_HOST }}
          CANFAR_SSH_PORT: ${{ secrets.CANFAR_SSH_PORT }}
        run: |
          mkdir -p ~/.ssh
          ssh-keyscan -p "${CANFAR_SSH_PORT:-64022}" -H "${CANFAR_SSH_HOST}" >> ~/.ssh/known_hosts 2>/dev/null || true
          {
            echo "Host ${CANFAR_SSH_HOST}"
            echo "  StrictHostKeyChecking no"
            echo "  UserKnownHostsFile /dev/null"
            echo "  IdentitiesOnly yes"
            echo "  Port ${CANFAR_SSH_PORT:-64022}"
          } >> ~/.ssh/config

      - name: Upload via SFTP (mkdir + put -r)
        env:
          CANFAR_USER: ${{ secrets.CANFAR_USER }}
          CANFAR_REMOTE_DIR: ${{ secrets.CANFAR_REMOTE_DIR }}
          CANFAR_SSH_HOST: ${{ secrets.CANFAR_SSH_HOST }}
          CANFAR_SSH_PORT: ${{ secrets.CANFAR_SSH_PORT }}
        run: |
          set -euo pipefail
          REMOTE_DIR="$CANFAR_REMOTE_DIR"
          PORT="${CANFAR_SSH_PORT:-64022}"
          HOST="$CANFAR_SSH_HOST"
          USER="$CANFAR_USER"
          STAGE="$GITHUB_WORKSPACE/canfar_sync"
          BATCH="/tmp/canfar_sftp_batch.txt"
          : > "$BATCH"
          # Build mkdir chain conditionally (sftp only; probe with ls)
          IFS='/' read -ra PARTS <<< "$REMOTE_DIR"
          ACC=""
          for part in "${PARTS[@]}"; do
            [ -z "$part" ] && continue
            ACC="${ACC}/${part}"
            if ! sftp -i ~/.ssh/id_canfar -P "$PORT" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -q -b - "$USER@$HOST" <<< "ls $ACC" >/dev/null 2>&1; then
              echo "mkdir $ACC" >> "$BATCH"
            fi
          done
          echo "cd $REMOTE_DIR" >> "$BATCH"
          echo "lcd $STAGE" >> "$BATCH"
          echo "put -r ." >> "$BATCH"
          sftp -i ~/.ssh/id_canfar -P "$PORT" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes -b "$BATCH" "$USER@$HOST"

README.md Filename collision: this file already exists in the selected repo/branch. Rename one of the files or the existing file will be overwritten. New file COMET will add to the repository. ? Project overview and quick-start instructions. Useful as a landing page for collaborators and future you. You can edit or replace it later.

# COMET: CANFAR tooling deployment

This repository is set up to deploy its contents to a CANFAR directory using GitHub Actions.

## How it works

- On pushes to your chosen branch, the workflow `.github/workflows/deploy-canfar.yml`:
  - Uses your repository secrets to establish SSH connectivity to CANFAR
  - Creates the destination directory if missing
  - Syncs your repository files (excluding `.git`) to `CANFAR_REMOTE_DIR`

## Required repository secrets

- `CANFAR_USER` — your CADC username used for CANFAR SSH
- `CANFAR_SSH_PRIVATE_KEY` — PEM private key generated in COMET (Ed25519)
- `CANFAR_REMOTE_DIR` — absolute path to deploy to (e.g., `/projects/CanDIAPL/HiPS`)
- `CANFAR_SSH_HOST` — usually `ws-uv.canfar.net`
- `CANFAR_SSH_PORT` — usually `64022`

## Customize

- Update the branch in the workflow if you prefer a branch other than `main`
- Add test gates, notifications, or scheduled syncs as needed

See the COMET docs for more guidance.

tests/test_comet.py Filename collision: this file already exists in the selected repo/branch. Rename one of the files or the existing file will be overwritten. New file COMET will add to the repository. ? Example test file so CI passes and you have a starting point. Use it to verify your environment and expand with real tests.

# Default pytest test generated by COMET
import pytest


def test_basic_math():
    # Keep a simple sanity check so CI passes by default
    assert 0.1 + 0.2 == pytest.approx(0.3)

Dockerfile Filename collision: this file already exists in the selected repo/branch. Rename one of the files or the existing file will be overwritten. New file COMET will add to the repository. ? Defines how to build your container image. Keep this if you want COMET to build and push a container image as part of deployment.

# Dockerfile (starter) generated by COMET
# Minimal Python container starter for astronomy / astrophysics projects
FROM python:3.11-slim

ENV PYTHONDONTWRITEBYTECODE=1     PYTHONUNBUFFERED=1     PIP_NO_CACHE_DIR=1     SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0+comet

WORKDIR /app

# Install a small scientific Python baseline; extend this as needed
RUN python -m pip install --upgrade pip &&     python -m pip install numpy pandas astropy

# Copy your repository content (adjust ignore rules in .dockerignore)
COPY . .

# If your repo defines extra dependencies, install them too. COMET sets a
# pretend VCS version because Docker build contexts normally omit .git.
# Editable package installation is best-effort so incomplete packaging metadata
# does not block a deployment-only starter image.
RUN if [ -f requirements.txt ]; then python -m pip install -r requirements.txt; fi &&     if [ -f pyproject.toml ] || [ -f setup.py ] || [ -f setup.cfg ]; then python -m pip install -e . || echo "Editable install failed; continuing with copied source."; fi

# Default command; change to your entry point/tool
CMD ["python"]

.dockerignore Filename collision: this file already exists in the selected repo/branch. Rename one of the files or the existing file will be overwritten. New file COMET will add to the repository. ? Lists files to exclude from the container build context. Keeps images smaller and avoids shipping secrets or bulky folders.

.git
.gitignore
__pycache__/
*.pyc
*.pyo
.venv/
venv/
env/
*.log
node_modules/
tests/
docs/
*.md
Dockerfile~
*.swp
*.swo

Sign in to a repository provider

GitHub

GitLab

Sign in to access CANFAR

Repository source

Choose deploy type

Choose template

Create a new repository

Use an existing repository

CANFAR

Secrets COMET will add to your repository:

Deploy

Secrets COMET will add to your repository:

Files COMET will commit:

Deployment Configured!

Your Setup Summary

What COMET commits for you

Sign in to a repository provider

GitHub

GitLab

Sign in to access CANFAR

Repository source

Choose deploy type

Choose template

Create a new repository

Use an existing repository

CANFAR

Secrets COMET will add to your repository:

Deploy

Secrets COMET will add to your repository:

Files COMET will commit:

Deployment Configured!

Your Setup Summary

Select CANFAR Directory

What COMET commits for you

Rename file

Add file

Upload file(s)

New Harbor repository

Generate unit tests

Delete file?