Skip to contents

Artalytics R Packages

Source: AGENTS.md

This document defines the shared expectations and tooling standards for all R packages in the Artalytics platform. It serves as a common reference for agentic tools (e.g., GitHub Copilot Agents, Sourcegraph, Cursor) to interact with each repository consistently and with full context. It covers both the internal utility libraries (like artcore, artutils, artopenai, etc.) and the Shiny module packages (like modBrowse, modUpload, modGallery, etc.) that together make up the Artalytics ecosystem.

Project Purpose and Ecosystem Overview

The Artalytics platform is composed of multiple R packages, each serving a specific role but adhering to common standards:

  • Core Utility Libraries (Internal): Packages such as artcore and artutils provide foundational functions and shared utilities for the platform. These are not directly user-facing but are consumed by other packages. artcore contains the lowest-level platform functions (database connections, CDN interfaces), while artutils builds on artcore to offer higher-level tools and data access functions used across modules and apps.

  • API Integration Libraries (Internal): Additional internal packages like artopenai (OpenAI integration), artgemini (Gemini integration), artsend (Resend email API), artopensea (OpenSea API), or artpixeltrace (artwork verification) extend functionality. They follow the same standards, ensuring they integrate seamlessly with core utilities.

  • UI Module Packages (Shiny-facing): Packages such as modArtist, modBrowse, modUpload, and modGallery implement Shiny modules (UI components + server logic) for the platform’s web interface. These modules are designed to be plugged into the main Artalytics Shiny application housed in appPlatform. They are developed as independent packages for modularity, but share the platform’s coding conventions and infrastructure.

  • Platform Application: The appPlatform package serves as the main parent application that orchestrates and integrates all mod* module packages into a cohesive user experience.

All these packages are versioned, documented, and tested consistently. These standards govern the full ecosystem, ensuring that whether a package is backend-focused or UI-facing, it conforms to the same high-level guidelines.

Package Dependency Hierarchy

The Artalytics platform follows a layered architecture with clear dependency boundaries:

┌─────────────────────────────────────────────────────────┐
│                     appPlatform                         │
│              (Main Shiny Application)                   │
├─────────────────────────────────────────────────────────┤
│   modArtist  │  modBrowse  │  modUpload  │  modGallery  │
│                    (Shiny Modules)                      │
├─────────────────────────────────────────────────────────┤
│  artopenai  │  artgemini  │  artpixeltrace  │  artcurator  │
│               (API & Domain Packages)                   │
├─────────────────────────────────────────────────────────┤
│                       artutils                          │
│           (High-level utilities & data access)          │
├─────────────────────────────────────────────────────────┤
│                        artcore                          │
│     (Low-level: DB connections, CDN interfaces)         │
└─────────────────────────────────────────────────────────┘

Key principles:

  • artcore provides the only functions allowed to open/close database connections (artcore::..dbc and artcore::..dbd) and the lowest-level CDN interface functions.
  • artutils builds on artcore to provide higher-level database query functions and CDN utilities that apps and modules rely on.
  • Modules and apps should ideally access data through artutils functions rather than calling artcore directly. The goal is to abstract artcore away from higher-level packages.

Note: During active development, these boundaries may be relaxed. For example, artcurator currently makes its own database queries. The long-term objective may involve consolidating all query logic into a dedicated package for true separation of concerns. This architecture is evolving.

Strict Dependency Rules

To maintain a clean, acyclic dependency graph, the following rules must be observed:

Package Layer Allowed Internal Dependencies
artcore None – must have zero Artalytics package dependencies
artutils artcore only
API/Domain packages (artopenai, artpixeltrace, etc.) artcore and/or artutils
Shiny modules (mod*) artutils (and by extension artcore), but never other modules
appPlatform All module packages and utilities as needed

Critical constraints:

  • artcore is dependency-free (internally): It should not import any other Artalytics-specific packages. It can rely on CRAN packages or third-party libraries, but it stands alone in the Artalytics ecosystem. This ensures artcore remains lightweight and universally usable.

  • artutils depends only on artcore: Among internal packages, artutils may only import artcore. External CRAN dependencies are permitted.

  • Modules must not import each other: Shiny modules are designed to be independent. If modules need to communicate or share state, that coordination happens via appPlatform’s main app logic—not by one module package importing another. This prevents circular dependencies and keeps modules independently testable.

Deployment Scope

This section clarifies how each type of package fits into deployment and the overall dependency structure.

Module Packages Are Not Standalone

Packages like modBrowse, modUpload, modGallery (and similar modules) are not standalone applications—they are components designed for integration into the main Artalytics Shiny app. Key implications:

  • Platform context assumed: Modules expect the presence of platform infrastructure (e.g., global options, artutils functions, shared reactive values). They will not function correctly if run outside of appPlatform.
  • No direct deployment: Modules are installed as dependencies of appPlatform, not deployed independently.
  • Development testing: Each module ships with an app.R file that wraps the module in a minimal parent structure for isolated development and testing. This is for development only—not production deployment.

Version Management

  • Version pinning: The main app’s (appPlatform) DESCRIPTION file pins compatible versions of module packages. When deploying, ensure module versions match the main app’s expectations.
  • Coordinated updates: If artcore is updated, dependent packages (artutils, modules, etc.) should be updated in tandem to remain compatible. Incompatible changes must be avoided or coordinated across the ecosystem.

Inter-Module Communication

Modules are designed to be independent and must not import each other. When modules need to communicate or share state:

  • Communication happens through appPlatform’s server logic
  • Shared data is passed via reactive values managed by the parent app
  • Global state or configuration is provided by artutils or app-level context

This design ensures modules remain independently testable and avoids circular or tangled dependencies.

Internal Utility Packages

artcore and artutils are core internal libraries that other packages depend on:

  • They are deployed as regular R packages (installable from GitHub)
  • They do not produce a direct user interface
  • They must be installed on any system running the Artalytics platform

Isolation for Testing

Each package can be loaded and tested in isolation:

  • artcore and artutils functions can be called by loading those packages alone
  • Shiny modules can be tested via their bundled app.R in a test context
  • Integration tests that verify cross-module behavior belong in appPlatform

Directory Structure

All Artalytics R packages use the same structure, with some modifications for Shiny app/module packages and API integration packages. Key aspects of the directory structure include:

  • R/ – Core R source code (functions, module definitions, etc.).
  • man/ – Documentation files (.Rd help files) generated by Roxygen2.
  • tests/ – Unit tests (testthat) in tests/testthat/ subdirectory.
  • inst/ – Additional files to install (e.g., inst/extdata for example data).
  • vignettes/ – Long-form docs/guides (often written in R Markdown or Quarto).
  • .github/workflows/ – CI workflow YAML files for GitHub Actions.

Shiny Module Packages:

In Shiny module packages, you will find additional components:

  • app.R – A standalone app file shipped with each module package for development and iteration. This app wraps the module in a minimal parent structure, allowing developers to test the module in isolation.
  • inst/www/ – Static assets for the module’s UI (images, JS, CSS) that are deployed with the package.
  • Quarto documents may appear in vignettes/ demonstrating module usage.

Database and CDN Access Patterns

Database Connections

Database connectivity follows a strict pattern:

  • Connection functions: Only artcore::..dbc() (connect) and artcore::..dbd() (disconnect) are permitted for database connections.
  • Query execution: Historically, only artutils functions were allowed to send queries and receive data. This rule has been relaxed during development (e.g., artcurator makes its own queries).
  • Session pattern: Shiny modules and apps typically make a database connection at session start to retrieve all data needed for the session. Exceptions include modUpload, which performs write/modify/delete operations based on user actions.

CDN Access

CDN (DigitalOcean Spaces) connectivity follows a similar layered pattern:

  • Low-level interface: artcore contains the CDN connection and interface functions.
  • High-level utilities: artutils provides higher-level CDN functionality that apps and modules use.
  • Objective: Apps and modules should use artutils CDN functions rather than calling artcore directly.

Documentation & Vignettes

All Artalytics packages share the same build process and documentation standards to ensure uniformity:

  • Documentation: Documentation is written as Roxygen2 comments for function files in R/*. These are generated into .Rd files in man/* using devtools::document(). All exported functions and user-facing objects must have Roxygen2 doc comments.

  • Vignettes: Packages can include vignettes (in vignettes/), written in R Markdown or Quarto, to provide tutorials or extensive examples. Call usethis::use_vignette("vignette_name") for setup. Use devtools::build_vignettes() to build them. Vignettes should be included in the package build so that users can access them via vignette("vignette_name").

  • R CMD check: Every package must pass R CMD check without errors or warnings. All examples, tests, and documentation are run/tested during the build. Developers are expected to run R CMD check (use devtools::build / devtools::check from R) locally before pushing changes. This ensures that all packages remain production-quality in terms of documentation and build integrity.

By adhering to these build and documentation standards, we guarantee that all packages have consistent, high-quality documentation and are built reproducibly. Agent tools can rely on Roxygen comments to provide in-context help, and contributors can expect a similar development experience across packages.

Testing Standards

All packages follow these testing practices:

  • Unit Testing (testthat): Use usethis::use_test() to setup testthat and devtools::test() (or testthat equivalents such as test_local).

  • Code Coverage (covr): Use the covr package to measure test coverage. Aim for high coverage (80%+) on critical functions. Coverage reports are generated locally with covr::package_coverage.

  • Tabular Datasets in R: Always use data.table as it’s highly efficient and its compact syntax is greatly preferred over data.frames, tibbles, and dplyr syntax. data.frames are strictly avoided.

  • Snapshot and UI Testing: For Shiny app and module packages, we utilize snapshotting features throughout the suite of testing packages:

    • Shiny modules: Use shinytest2 framework to simulate a Shiny session and verify module behavior (inputs lead to correct outputs or UI state).
    • API packages: Use mockery or httptest2 for mocking external API calls.
    • General snapshots: Use testthat::expect_snapshot() to capture function output for stability across changes.

Testing Scope

  • Module packages (mod*): Tests should focus on the module in isolation. Each module package ships with an app.R file that wraps the module for development testing.
  • Integration tests: Cross-module and full application integration tests belong in appPlatform, not in individual module packages.
  • Exported functions only: Only exported package functions should have corresponding unit tests. Internal functions are typically covered by testing their parent function callers. This keeps test code compact, manageable, and robust.

Style Guide and Coding Conventions

We maintain a consistent coding style across all R packages to make the codebase easy to read and navigate. Key style guidelines include:

  • Favor data.table over tidyverse: We prefer using data.table for data manipulation rather than dplyr or other tidyverse packages. This means using data.table’s syntax (DT[i, j, by] and chaining with []) for subsetting and aggregation. The codebase avoids attaching the tidyverse; instead it leans on base R and data.table for performance and clarity. This also reduces external dependencies.

  • Native Pipe Operator: All packages that utilize the pipe operator must use R’s native pipe |> (introduced in R 4.1+).

  • String Handling: We prefer stringr for string operations (excluding concatenation, see next) over base R string functions. For example, use stringr::str_detect, str_replace, etc., instead of base R’s grepl, gsub, etc. This provides more readable and robust regex handling. All string constants should be written clearly, and where patterns are complex, commented for clarity.

  • Concatenation: Use paste0() instead of paste() when concatenating strings without a separator. In general, be mindful of unnecessary whitespace or separators in output. Consistency in constructing strings (especially for file paths or messages) makes the behavior predictable.

  • Code Conciseness: Aim to write code that is compact. Implementations that meet all requirements while reducing lines of code are always preferred (all else equal). Shiny app code has the reputation of quickly becoming too large and challenging to manage as features are added and tech debt grows.

    For example: data.table is strictly favored for its efficiency, and it comes with the added benefit of having highly compact syntax. When used in conjunction with a naming convention that prioritizes more comments but fewer character names, we can produce an elegant codebase that can be frequently de-risked of tech debt—ensuring quality standards are always adhered to.

  • Function Documentation and Export: All exported functions (and any internal ones that are complex) should have roxygen2 documentation blocks. This includes description, parameter documentation, return value, examples, and any @seealso or @references if relevant. This allows tools and contributors to quickly grasp usage. Internal (non-exported) functions can have shorter comments if needed, but key ones often get a @noRd roxygen comment for clarity in the code.

  • Linting: We use lintr in CI to enforce a basic style check on all packages. Developers should strive to write code that passes lintr::lint_package() while ensuring specific organization-adopted styling is maintained. Note that our style leans toward base R and data.table usage, which is compatible with the default lintr settings.

  • Naming Conventions: Use snake_case for function and variable names (consistent with base R and tidyverse conventions). Filenames in R/ generally match the name of the primary function or concept defined in the file (to ease navigation).

CI/CD and GitHub Actions

Continuous Integration (CI) and Continuous Deployment/Development workflows are set up for each package to maintain quality and consistency:

  • GitHub Actions Workflows: Each repository contains YAML files under .github/workflows/ that automate build and test processes. At minimum, there is a workflow for running R CMD check (which includes tests) and a workflow for linting. Many packages also have a test coverage workflow and possibly a deploy or pkgdown documentation workflow.

    • The R CMD check workflow installs the package’s dependencies (including other Artalytics packages from GitHub if listed in Remotes) and runs R CMD check on the package across one or more OS (commonly Ubuntu, and sometimes Windows/macOS as needed). It uses the standard r-lib/actions for consistency.
    • The Lint workflow runs lintr::lint_package() to ensure coding standards are met (without failing the build on style issues as of now).
    • The Coverage workflow (if present) uses covr to run tests and upload coverage results to a service like Codecov.

  • Quality Gates: A pull request must pass R CMD check (which implies all tests passed) and linting before it can be merged. This gate keeps the main branch stable and clean.

  • Secrets & Private Dependencies: Many Artalytics packages depend on other internal packages (not on CRAN). To allow CI to install these, the GitHub Org secret GITHUB_PAT is injected into the workflow environment. Setting this during the build process lets pak::pkg_install("artalytics/{repo_name}") install the private package. Similarly, any API keys or tokens needed for tests are provided via secrets:

    • For example, test coverage upload uses a CODECOV_TOKEN secret (set in the repo settings) so that covr::codecov() can report coverage.
    • If a package needed an API key (say for OpenAI), it would be set as an env var (e.g., ART_OPENAI_KEY) in the workflow, rather than hardcoded.

All CI/CD workflows are kept similar across repos, which means an agent or developer can look at the .github/workflows/ folder and navigate them easily. If you’re in one package, you can expect the same checks and balances to apply in another, fostering a unified development workflow.

Security and Secret Handling

Security best practices are critical in our development process. All packages follow these rules regarding secrets and sensitive information:

  • Env Vars for Secrets: Store secrets in .Renviron for development/iteration and never commit this file. In production or CI, these are set through secure means (never hardcoded).

  • CI Secret Injection: In CI workflows, any required secrets (PATs, tokens) are added via GitHub Actions secrets. As noted, the GITHUB_PAT for installing private packages (pak::pkg_install("{org}/{repo}")).

  • Runtime Config: Artalytics platform packages and apps utilize certain standard environment variables at runtime across packages.

Environment Variables Reference

The following environment variables are used across the platform:

Application Mode

Variable Description Example Value
ART_RUNAS_DEMO Disables all UI actions that write/delete/modify data across any source (database, CDN, etc.). Used for deploying demo applications. This will be phased out as authentication/login is rolled out. TRUE

Database Configuration

Used in artcore for creating connections managed by artutils query functions:

Variable Description Example Value
ART_USE_CONFIG Toggle database context artdev
ART_PGHOST_DEV Managed DB host art-dev-do-user-....ondigitalocean.com
ART_PGPORT_DEV Database port 25060
ART_PGUSER_DEV Dev database user shiny
ART_PGPASS_DEV Dev user password (set in .Renviron)

API Integrations

Variable Required By Description
ART_OPENAI_KEY artopenai OpenAI API key
ART_GEMINI_KEY artgemini Google Gemini API key
ART_RESEND_KEY artsend Resend email API key
ART_OPENSEA_KEY artopensea OpenSea API key

CDN (DigitalOcean Spaces)

Variable Description
ART_BUCKETS_KEY_ID Spaces access key ID
ART_BUCKETS_KEY_SECRET Spaces secret access key

Appendix

Repository Reference

Most Artalytics repos are private; programmatic access requires your GITHUB_PAT. Public entries are marked “public” and don’t need the token.

Repository Description Visibility
modFrames Artalytics Platform Shiny Module for Artwork-Creation (Replay) Analytics. private
artutils Package containing tools and data shared across the modules of the Artalytics Shiny app private
appPlatform Artalytics Platform Main App that Wraps All Shiny Modules into a Single Parent Application. private
artpipelines Pipelines that generate data needed by the Artalytics platform private
modGallery Artalytics Platform Shiny Module Containing Digital Gallery Experiences for Authenticated Artwork Projects. private
modUpload Artalytics Platform Shiny Module for Uploading/Managing Artwork Projects. private
modBrowse Artalytics Platform Shiny Module for Browsing Artworks and Collections. private
artbenchmark Platform Tools for Robust Artwork Valuation and Performance/Price Benchmarking of Authenticated Works. private
artpixeltrace Artwork verification and certificate generation tools used by the Artalytics Platform. private
artopensea Tools for an OpenSea integration used by the Artalytics App Platform. private
artopenai Tools for an OpenAI integration used by the Artalytics App Platform. private
artcore Core Tools Required Across Platform R Packages, Shiny Modules, and Deployed Applications. private
modArtist Artist Profile is a module of the artalytics platform and serves as the entry point to the app. Provides complete user gallery experience: Learn about Artist → Browse their collections → Explore an artwork → Dive deeper with replay analytics → Explore the Artist’s stores private
artsend API integration package enabling resend functionality throughout the platform. private
artauth Authentication and access control for Artalytics platform - passwordless magic links, user account management, role-based access (artists, collectors, investors), and lead capture functionality private
arthelpers Internal development-only toolkit to address codebase/data management, architecture audits, debugging/feature development utils. private
artshiny Shared Shiny UI components for Artalytics platform private
artcurator Package containing tools for retrieval of data, memory, and database plus s3 bucket context, specifically to enable an integrated layer for CuratorAI - an AI chatbot living inside the Artalytics web app that understands an artist, an artwork, the metadata, the provenance, valuations, timelines, collection context, etc private
artgemini R package containing API client functions and tools related to Google’s Gemini AI Platform API. private

TODO Summary

The following areas require additional documentation:

  1. artshiny package - Document shared UI components and utilities
  2. Database query separation - Finalize and document target architecture
  3. CDN access patterns - Document artcore/artutils CDN functions and usage patterns
  4. Testing examples - Add reproducible examples for shinytest2, httptest2, snapshots
  5. Style guide examples - Add concrete code examples for preferred patterns
  6. Environment variable helper - Document internal pattern for retrieving/validating env vars