diff --git a/AGENTS.md b/AGENTS.md index 1b27081..0e0c9ed 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,60 @@ # AGENTS.md +## Mandatory security and URL contracts + +- URL クエリへの JSON 格納を禁止する。Base64・16進数等による迂回も禁止。 + 単純な配列は半角スペース区切り、構造化条件は番号付きの個別キーで表す。 + 未指定と明示的な空選択を区別し、自由入力を区切り文字で分割しない。 + Frontend / Backend で同じ文法・上限・検証を使い、新しい URL 状態の追加前に確認する。 +- 一度除票されたリソースの除票以前（除票版を含む）の履歴は admin 以外へ公開しない。 + 恢復しても解除しない。すべての後続除票を確定版番号と正しい複合キーで判定する。 + 本文・属性・ID・版番号・関連参照・旧値を API / SSR / キャッシュ / 出力から漏らさない。 + サーバーが確定した current_user の role のみを認可に使用する。 + 秘匿版も安全な全体件数には含めるが、公開明細・内訳・検索件数とは分離する。 + Counts では、現在の条件に対する権限除外前の出力行数と権限除外レコード数の + 数値公開及びその差分による間接的な情報開示を許容する。これは件数だけの例外であり、 + 秘匿された本文・属性・識別情報・集計ラベルを公開する例外ではない。 + それ以外の経路で秘匿属性を条件・軸・ソート・差分から推測できる実装は禁止する。 + +## Mandatory Counts data contract + +- Counts の取得対象の ID・集合・件数・並び順を UI ロカールに依存させない。 + 言語・文字体系を抽出条件に使う場合は明示的な項目・条件として扱う。 +- API は記録済みの全言語名と別名を保持する。履歴の名前を現在の名前で置換しない。 + 名称・別名列は Language の言語コードから一言語一列で定義し、Locale ごとに増やさない。 + 同じ言語の全文字体系を含め、script_code 等の記録属性を保持する。 + 未登録値を別言語で補完しない。旧ロカール限定条件の互換処理では意味を無言で変えない。 + 初期表示列だけは UI ロカールに合わせてよい。明示的な列指定・空選択を優先する。 +- 明細の内表は行集合を保持し、表示列・列順・列幅だけを表示設定として操作する。 + 内表行を抽出・除外する表示機能を追加しない。内表を参照する抽出条件は外表の + 採否だけを決定し、一致した内表行だけを残す実装は禁止する。 +- 言語名はタグ一件の言語別列とし、言語名のために内表をもう一段増やさない。 + 複数の別名は欠落させず、複数値の項目又は別名の内表として保持する。 +- JSON は対象タグの全言語登録値を保持する。CSV は選択された表示列を出力する。 + UI 翻訳済みラベルでデータをソート・識別したり、JSON の項目キーを変えたりしない。 + +## Mandatory naming contract + +- メソッド・関数など、実際に処理を実行する callable の名前以外の自前識別子に + 動詞を使用しない。状態は名詞・形容詞・形容詞的な語で表す。今後の全開発に適用する。 +- 変数、引数、プロパティ、型定義内の項目、API・URL パラメータ、JSON キー、 + DB カラムにも適用する。`is_` / `is`、`has_` / `has`、`can_` / `can`、 + `should_` / `should` などの動詞・助動詞由来の接頭辞を使用しない。 + 状態を格納する識別子に `show`、`enable` 等の動詞も使用しない。 +- 形容詞で状態を表せる場合はその語を使う。例：`is_latest` / `isLatest` は + `latest`、`is_active` は `active`、`is_deleted` は `deleted`、 + `is_visible` は `visible`、表示状態は `visible`、有効状態は `enabled`。 +- 名詞で真偽値を表す場合は原則 `_flg`、camelCase では `Flg` を付ける。 + 例：`is_admin` は `admin_flg`、`is_owner` は `owner_flg`、 + `isAdmin` は `adminFlg`。クエリパラメータでは意味が明確なら接尾辞を省略できる。 +- callable の `setState`、`onChange` 等や、外部ライブラリ・フレームワーク所定の + API 名は変更しない。外部値を自前の状態へ保持する場合は境界で適切な別名を付ける。 + 言語固有の表記規約を維持し、予約語や曖昧さは規則を満たす明確な代替名で避ける。 +- 改名時は Frontend / Backend、URL、API、出力の名称を整合させる。 + 旧 URL・API の互換性は入出力境界で扱い、旧命名を新しい内部コードへ伝播させない。 + DB の改名は既存データを保持する追加マイグレーションで扱い、適用済みのものを + 書き換えない。メソッド名・外部 API・文字列値等を機械的な一括置換で破壊しない。 + ## Project overview BTRC Hub / タグ広場 is a split Rails API and React frontend repository. @@ -573,10 +628,10 @@ and layout reuse, follow `frontend/AGENTS.md`. - When new user-facing wording appears necessary, ask the user for the exact wording and placement before implementing it. - Do not invent replacement copy when removing unrequested wording. -- Do not create, modify, or run tests unless the user explicitly asks for - test work. When the user asks for tests, keep working within the permitted - test-file scope and rerun them until they pass or the remaining failure is - clearly blocked. +- 明示的な個別指示がない限り、テストの追加・修正・削除・実行を一切行わない。 + fixture、mock、snapshot、テスト補助コードも対象とする。 + 既存のテスト差分を勝手に修正・差戻ししない。 + この規約はディレクトリ固有の自動的なテスト実行推奨より優先する。 - Test-only work includes adding, updating, deleting, reorganising, or fixing SyntaxError in tests. During test-only work, do not modify production code. - During test-only work, do not change production constants, behaviour, API @@ -631,6 +686,19 @@ and layout reuse, follow `frontend/AGENTS.md`. ## Frontend rules +- 原則として、ユーザ向け UI 文言を TSX / TypeScript に直接記述しない。 + 新しい文言は既存の i18n システムで管理し、適切な既存キーを先に確認する。 + 不必要な重複定義を避け、新規機能固有の文言は適切な namespace にまとめる。 +- 対応 locale の辞書を整合させる。未翻訳項目には既存の日本語 fallback と + `_untranslated` 規約を使い、language metadata を維持する。 + 翻訳を捏造せず、未翻訳文字列を正式な翻訳として登録しない。 +- UI 文言の追加・変更時はビルド、静的検査、実画面の確認等で、辞書と + SSR / hydration の整合及び翻訳キーの非露出を検証する。 + i18n の検証を理由にテスト関連作業を開始しない。 +- API の内部識別子、パラメータ、ユーザ入力データ、外部 URL は機械的に + 翻訳しない。ボタン、項目名、CSV ヘッダ、エラー、空状態、ツールチップ、 + アクセシビリティ用ラベルも UI 文言として扱う。 + - Use `frontend/src/lib/api.ts` for API calls so headers and camelCase conversion stay consistent. - Add or reuse TanStack Query keys through `frontend/src/lib/queryKeys.ts`; avoid ad hoc query key arrays. diff --git a/backend/AGENTS.md b/backend/AGENTS.md index e0f304e..e476fbd 100644 --- a/backend/AGENTS.md +++ b/backend/AGENTS.md @@ -1,5 +1,43 @@ # backend/AGENTS.md +## Mandatory security and URL contracts + +- URL クエリへの JSON 格納を禁止する。Base64・16進数等による迂回も禁止。 + 単純な配列は半角スペース区切り、構造化条件は番号付きの個別キーで表す。 + 未指定と明示的な空選択を区別し、自由入力を区切り文字で分割しない。 + Frontend / Backend で同じ文法・上限・検証を使い、新しい URL 状態の追加前に確認する。 +- 一度除票されたリソースの除票以前（除票版を含む）の履歴は admin 以外へ公開しない。 + 恢復しても解除しない。すべての後続除票を確定版番号と正しい複合キーで判定する。 + 本文・属性・ID・版番号・関連参照・旧値を API / SSR / キャッシュ / 出力から漏らさない。 + サーバーが確定した current_user の role のみを認可に使用する。 + 秘匿版も安全な全体件数には含めるが、公開明細・内訳・検索件数とは分離する。 + Counts では、現在の条件に対する権限除外前の出力行数と権限除外レコード数の + 数値公開及びその差分による間接的な情報開示を許容する。これは件数だけの例外であり、 + 秘匿された本文・属性・識別情報・集計ラベルを公開する例外ではない。 + それ以外の経路で秘匿属性を条件・軸・ソート・差分から推測できる実装は禁止する。 +- 履歴読み取りは HistoryAccess の共通認可を必須とする。書込用 relation を公開しない。 + export は生成時と取得時に認可し、権限変更・除票後に古いファイルを再利用しない。 + +## Mandatory Counts data contract + +- Counts のロカール非依存・内表行集合保持はルート AGENTS.md の必須契約に従う。 + UI 言語を名前の WHERE・ORDER BY へ暗黙に使わない。名前検索は記録済みの + 全言語又は明示された言語・文字体系を使う。関連名にも HistoryAccess を適用する。 + JSON は全言語名を保持し、CSV の列選択を理由に元スナップショットを削らない。 + 名称・別名列は Language から生成し、Locale の script_code で同じ言語の値を落とさない。 + 旧ロカール列の条件互換は入力境界に限定し、旧キーの文字体系限定を維持する。 + +## Mandatory naming contract + +- ルート `AGENTS.md` の必須命名規約を適用する。メソッド等の callable 以外の + 自前識別子に動詞を使わず、形容詞の状態名、名詞の真偽値には `_flg` を使用する。 + 例：`is_latest` は `latest`、`is_admin` は `admin_flg`。 +- 引数、属性、SQL alias、API・URL パラメータ、JSON キー、DB カラムも対象とする。 + Rails 等が定義する外部 API 名や、処理を実行するメソッド名は変更しない。 +- 旧 API・URL の互換性は解析・シリアライズ境界で扱い、内部定義は新名に統一する。 + DB カラムの改名はデータを保持する追加マイグレーションで設計し、適用済みの + マイグレーションを書き換えない。マイグレーションの実行には個別指示を要する。 + ## Scope These rules apply to work under `backend/`. diff --git a/backend/app/controllers/application_controller.rb b/backend/app/controllers/application_controller.rb index 4f2610f..3e9305a 100644 --- a/backend/app/controllers/application_controller.rb +++ b/backend/app/controllers/application_controller.rb @@ -1,4 +1,5 @@ class ApplicationController < ActionController::API + include RequestAccess include ActionController::Cookies include ActionController::RequestForgeryProtection @@ -8,34 +9,49 @@ class ApplicationController < ActionController::API rescue_from ActiveRecord::RecordInvalid, with: :render_record_invalid rescue_from ActiveRecord::RecordNotUnique, with: :render_record_not_unique + rescue_from(ActiveRecord::RecordNotFound) { head :not_found } rescue_from(UnsupportedLocale) { head :not_found } before_action :reject_banned_ip_address! before_action :authenticate_user before_action :reject_banned_user! before_action :validate_explicit_locale! + before_action :enforce_query_contract! + around_action :with_history_authority + rescue_from(HistoryAccess::Forbidden) { head :not_found } protect_from_forgery with: :exception self.forgery_protection_origin_check = true rescue_from ActionController::InvalidAuthenticityToken, with: :reject_invalid_csrf after_action :prevent_authenticated_caching - def current_user = @current_user - private + def enforce_query_contract! + if request.query_parameters.keys.any? { _1 == 'tag_refs' || _1.start_with?('tag_refs.') } + return head :bad_request + end + pairs = URI.decode_www_form(request.query_string) + return head :bad_request if pairs.length > 1000 || pairs.map(&:first).uniq.length != pairs.length + return head :bad_request if request.query_parameters.values.any? { !_1.is_a?(String) } + if (controller_name.start_with?('counts_') && + (request.query_parameters.keys & ['headers', 'labels']).any?) + return head :bad_request + end + rescue ArgumentError + head :bad_request + end + + def with_history_authority(&action) + response.headers['Cache-Control'] = 'private, no-store' + HistoryAccess.set(user: current_user, &action) + end + def reject_invalid_csrf response.headers['X-CSRF-Invalid'] = '1' response.headers['Cache-Control'] = 'private, no-store' head :forbidden end - def authenticate_user - code = session[:inheritance_code] - return if code.blank? - - @current_user = User.find_by(inheritance_code: code) - end - def prevent_authenticated_caching response.headers['Cache-Control'] = 'private, no-store' if current_user end @@ -79,19 +95,6 @@ class ApplicationController < ActionController::API render_validation_error base: ['すでに存在してゐます．'] end - def reject_banned_ip_address! - ip_address = IpAddress.find_by(ip_address: IPAddr.new(request.remote_ip).hton) - return unless ip_address&.banned? - - head :forbidden - end - - def reject_banned_user! - return unless current_user&.banned? - - head :forbidden - end - def render_validation_error record = nil, fields: { }, base: [], status: :unprocessable_entity errors = { } diff --git a/backend/app/controllers/concerns/counts_listing.rb b/backend/app/controllers/concerns/counts_listing.rb new file mode 100644 index 0000000..3b1c03e --- /dev/null +++ b/backend/app/controllers/concerns/counts_listing.rb @@ -0,0 +1,21 @@ +module CountsListing + extend ActiveSupport::Concern + + def index + locale = resolve_locale! + resource = is_a?(CountsPostsController) ? 'posts' : 'tags' + query = CountsQuery.new(version_scope, resource, request.query_parameters.except('locale') + .merge('locale' => locale.code)) + result = { population_count_scope: 'all_recorded_versions', population_count: HistoryAccess.population_count(query.relation.klass), count: query.count, page: query.page, limit: query.limit, + at: request.query_parameters['at'].presence || Time.current.utc.iso8601(6) } + if query.mode == 'summary' + result.merge!(query.summary) + else + result[:versions] = CountsVersionRepr.many( + query.details, resource:, language_code: locale.language_code, history_flg: true) + end + render json: result + rescue ArgumentError, TypeError + head :bad_request + end +end diff --git a/backend/app/controllers/concerns/history_storage_access.rb b/backend/app/controllers/concerns/history_storage_access.rb new file mode 100644 index 0000000..69ca253 --- /dev/null +++ b/backend/app/controllers/concerns/history_storage_access.rb @@ -0,0 +1,52 @@ +# Permanent storage URLs must not bypass a resource's later discard boundary. +module HistoryStorageAccess + extend ActiveSupport::Concern + + included do + include RequestAccess + prepend_before_action :authorise_history_storage! + end + + private + + def authorise_history_storage! + response.headers['Cache-Control'] = 'private, no-store' + return unless request.get? || request.head? + + reject_banned_ip_address! + return if performed? + + authenticate_user + reject_banned_user! + return if performed? + + signed = params[:signed_blob_id] || params[:signed_id] + blob = if signed + ActiveStorage::Blob.find_signed(signed) + elsif params[:encoded_key] + payload = ActiveStorage.verifier.verified(params[:encoded_key], purpose: :blob_key) + key = payload && (payload[:key] || payload['key']) + if key + original = key.start_with?('variants/') ? key.split('/')[1] : key + ActiveStorage::Blob.find_by(key: original) + end + end + return head :not_found unless blob + + allowed = HistoryAccess.set(user: current_user) { HistoryAccess.blob_visible?(blob) } + head :not_found unless allowed + rescue ActiveSupport::MessageVerifier::InvalidSignature + head :not_found + end + + # Override the engine's public-forever caching before a streaming response commits. + def http_cache_forever(**_options) + response.headers['Cache-Control'] = 'private, no-store' + yield + end + + def expires_in(*arguments, **options) + super + response.headers['Cache-Control'] = 'private, no-store' + end +end diff --git a/backend/app/controllers/concerns/request_access.rb b/backend/app/controllers/concerns/request_access.rb new file mode 100644 index 0000000..4d7cf6a --- /dev/null +++ b/backend/app/controllers/concerns/request_access.rb @@ -0,0 +1,28 @@ +# Canonical read-side session and BAN boundary, shared by API and storage controllers. +module RequestAccess + extend ActiveSupport::Concern + + def current_user = @current_user + + private + + def authenticate_user + code = session[:inheritance_code] + return if code.blank? + + @current_user = User.find_by(inheritance_code: code) + end + + def reject_banned_ip_address! + ip_address = IpAddress.find_by(ip_address: IPAddr.new(request.remote_ip).hton) + return unless ip_address&.banned? + + head :forbidden + end + + def reject_banned_user! + return unless current_user&.banned? + + head :forbidden + end +end diff --git a/backend/app/controllers/counts_exports_controller.rb b/backend/app/controllers/counts_exports_controller.rb new file mode 100644 index 0000000..dff8f05 --- /dev/null +++ b/backend/app/controllers/counts_exports_controller.rb @@ -0,0 +1,185 @@ +class CountsExportsController < ApplicationController + wrap_parameters false + + def create_job + locale = resolve_locale! + payload = request.request_parameters.merge(request.query_parameters) + session[:counts_export_owner] ||= SecureRandom.hex(32) + job = CountsExportJobs.start(payload, session[:counts_export_owner], locale.language_code) + response.headers['Cache-Control'] = 'private, no-store' + render json: { job: }, status: :accepted + rescue CountsExportJobs::WorkerUnavailable + render json: { error: 'export_worker_unavailable' }, status: :service_unavailable + rescue CountsExportDownload::Busy + head :too_many_requests + rescue CountsAxes::TooManyAxes + render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request + rescue CountsAxes::PrivateCondition + render json: { error: 'private_condition' }, status: :forbidden + rescue CountsTableQuery::TooManyColumns + render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS }, + status: :unprocessable_entity + rescue ActiveRecord::StatementTimeout + render json: { error: 'counts_query_timeout' }, status: :service_unavailable + rescue ArgumentError, TypeError, KeyError, JSON::ParserError + head :bad_request + end + + def job_status + job_state(cancel: false) + end + + def cancel_job + job_state(cancel: true) + end + + def start_download + result = CountsExportJobs.download_attempt(params[:token], session[:counts_export_owner]) + return head :not_found unless result + + response.headers['Cache-Control'] = 'private, no-store' + render json: result + end + + def prepare + locale = resolve_locale! + payload = request.request_parameters.merge(request.query_parameters) + exporter = CountsExporter.new(payload, language_code: locale.language_code) + session[:counts_export_owner] ||= SecureRandom.hex(32) + token = CountsExportDownload.prepare(exporter, session[:counts_export_owner]) + response.headers['Cache-Control'] = 'private, no-store' + render json: { token: } + rescue CountsExportDownload::Busy + head :too_many_requests + rescue CountsExportDownload::TooLarge + head 413 + rescue CountsExportDownload::TimedOut + head :gateway_timeout + rescue CountsExportDownload::StorageFull, Errno::ENOSPC + head :insufficient_storage + rescue CountsAxes::TooManyAxes + render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request + rescue CountsAxes::PrivateCondition + render json: { error: 'private_condition' }, status: :forbidden + rescue CountsTableQuery::TooManyColumns + render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS }, + status: :unprocessable_entity + rescue ActiveRecord::StatementTimeout + render json: { error: 'counts_query_timeout' }, status: :service_unavailable + rescue ArgumentError, TypeError, KeyError, JSON::ParserError + head :bad_request + end + + def download + if params[:job].present? || params[:attempt].present? + return head :method_not_allowed unless request.get? + + return confirmed_download + end + file = CountsExportDownload.find(params[:token], session[:counts_export_owner]) + return head :not_found unless file + + response.headers['Cache-Control'] = 'private, no-store' + send_file file.fetch('path'), + filename: "counts-#{ file.fetch('resource') }.#{ file.fetch('format') }", + type: file['format'] == 'json' ? 'application/json' : 'text/csv', + disposition: 'attachment' + end + + def index + locale = resolve_locale! + payload = request.request_parameters.except('authenticity_token') + .merge(request.query_parameters) + exporter = CountsExporter.new(payload, + language_code: locale.language_code) + session[:counts_export_owner] ||= SecureRandom.hex(32) + token = CountsExportDownload.prepare(exporter, session[:counts_export_owner]) + file = CountsExportDownload.find(token, session[:counts_export_owner]) + return head :not_found unless file + + send_file file.fetch('path'), + filename: "counts-#{ exporter.resource }.#{ exporter.format }", + type: exporter.format == 'json' ? 'application/json' : 'text/csv', + disposition: 'attachment' + rescue CountsAxes::TooManyAxes + render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request + rescue CountsAxes::PrivateCondition + render json: { error: 'private_condition' }, status: :forbidden + rescue CountsTableQuery::TooManyColumns + render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS }, + status: :unprocessable_entity + rescue ActiveRecord::StatementTimeout + render json: { error: 'counts_query_timeout' }, status: :service_unavailable + rescue ArgumentError, TypeError, KeyError, JSON::ParserError + head :bad_request + end + + private + + def confirmed_download + owner = session[:counts_export_owner] + arguments = { file_token: params[:token], attempt: params[:attempt] } + if (!(arguments[:attempt].present?) || + !(CountsExportJobs.download_attempt(params[:job], owner, **arguments))) + return head :not_found + end + + file = CountsExportDownload.find(params[:token], owner) + unless file + CountsExportJobs.download_attempt(params[:job], owner, **arguments, status: 'unavailable') + return head :not_found + end + job = params[:job] + authority = HistoryAccess.user + first_check = true + state = lambda do |status| + if status == 'active' + if first_check + first_check = false + fresh = authority && User.find_by(id: authority.id) + next false if authority && (!fresh || fresh.banned?) + + authority = fresh + HistoryAccess.set(user: authority) do + CountsExportDownload.find(arguments[:file_token], owner) && + CountsExportJobs.stream_active?(job, owner, **arguments) + end + else + CountsExportJobs.stream_active?(job, owner, **arguments) + end + else + HistoryAccess.set(user: authority) do + CountsExportJobs.download_attempt(job, owner, **arguments, status:) + end + end + end + unless CountsExportJobs.download_attempt(job, owner, **arguments, status: 'streaming') + return head :conflict + end + response.headers['Cache-Control'] = 'private, no-store' + response.headers['Last-Modified'] = File.mtime(file.fetch('path')).httpdate + response.headers['X-Accel-Buffering'] = 'no' + send_file file.fetch('path'), + filename: "counts-#{ file.fetch('resource') }.#{ file.fetch('format') }", + type: file['format'] == 'json' ? 'application/json' : 'text/csv', + disposition: 'attachment' + # Keep Rails' bounded FileBody reader, but hide to_path from Rack::Sendfile: + # offloading would bypass the GET acknowledgement and cancellation checks. + stream = CountsExportDownload::ResponseBody.new(response.stream, &state) + self.response_body = stream + rescue IOError, SystemCallError + stream&.close + CountsExportJobs.download_attempt(params[:job], owner, **arguments, status: 'failed') + raise if response.committed? + + head :service_unavailable + end + + def job_state cancel: + result = CountsExportJobs.status(params[:token], session[:counts_export_owner], cancel:) + return head :not_found unless result + + response.headers['Cache-Control'] = 'private, no-store' + render json: result + end +end diff --git a/backend/app/controllers/counts_posts_controller.rb b/backend/app/controllers/counts_posts_controller.rb new file mode 100644 index 0000000..74ed700 --- /dev/null +++ b/backend/app/controllers/counts_posts_controller.rb @@ -0,0 +1,3 @@ +class CountsPostsController < PostVersionsController + include CountsListing +end diff --git a/backend/app/controllers/counts_states_controller.rb b/backend/app/controllers/counts_states_controller.rb new file mode 100644 index 0000000..1091bf1 --- /dev/null +++ b/backend/app/controllers/counts_states_controller.rb @@ -0,0 +1,10 @@ +class CountsStatesController < ApplicationController + def index + locale = resolve_locale! + result = CountsStateQuery.new(request.query_parameters, + language_code: locale.language_code).call + render json: result + rescue ArgumentError, TypeError + head :bad_request + end +end diff --git a/backend/app/controllers/counts_tables_controller.rb b/backend/app/controllers/counts_tables_controller.rb new file mode 100644 index 0000000..6a6f5bb --- /dev/null +++ b/backend/app/controllers/counts_tables_controller.rb @@ -0,0 +1,47 @@ +class CountsTablesController < ApplicationController + def definition + locale = resolve_locale! + definition = CountsResourceDefinition.new(params[:resource]) + render json: { fields: definition.metadata, + output_fields: definition.output_fields, + filter_fields: definition.filter_metadata, + axes: { latest: definition.axes(false), history: definition.axes(true) }, + name_columns: definition.name_columns, + column_aliases: definition.column_aliases, + condition_aliases: definition.condition_aliases, + grains: CountsAxes::GRAINS, + defaults: CountsDefaults.metadata(params[:resource], locale: locale.code) } + rescue ArgumentError + head :bad_request + end + + def index + resolve_locale! + render json: CountsTableQuery.new(request.query_parameters).call + rescue CountsAxes::TooManyAxes + render json: { error: 'too_many_axes', max_axes: CountsAxes::MAX_AXES }, status: :bad_request + rescue CountsAxes::PrivateCondition + render json: { error: 'private_condition' }, status: :forbidden + rescue CountsTableQuery::TooManyColumns + render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS }, + status: :unprocessable_entity + rescue CountsPivotTotals::TimedOut + Rails.logger.warn('counts_query_timeout: pivot_totals') + render json: { error: 'counts_query_timeout', stage: 'totals', + timeout_seconds: CountsTableQuery::SQL_TIMEOUT_SECONDS }, + status: :service_unavailable + rescue ActiveRecord::StatementTimeout => error + Rails.logger.warn("counts_query_timeout: #{ error.class }") + render json: { error: 'counts_query_timeout' }, status: :service_unavailable + rescue ArgumentError, TypeError, KeyError, JSON::ParserError + head :bad_request + end + + def candidates + resolve_locale! + render json: CountsCandidates.new(request.query_parameters.merge( + 'resource' => request.path_parameters[:resource])).call + rescue ArgumentError, TypeError, KeyError + head :bad_request + end +end diff --git a/backend/app/controllers/counts_tags_controller.rb b/backend/app/controllers/counts_tags_controller.rb new file mode 100644 index 0000000..2bda72a --- /dev/null +++ b/backend/app/controllers/counts_tags_controller.rb @@ -0,0 +1,3 @@ +class CountsTagsController < TagVersionsController + include CountsListing +end diff --git a/backend/app/controllers/material_versions_controller.rb b/backend/app/controllers/material_versions_controller.rb index 50f59fa..9ee5ff1 100644 --- a/backend/app/controllers/material_versions_controller.rb +++ b/backend/app/controllers/material_versions_controller.rb @@ -8,7 +8,7 @@ class MaterialVersionsController < ApplicationController offset = (page - 1) * limit - q = MaterialVersion.includes(:created_by_user) + q = HistoryAccess.visible(MaterialVersion.includes(:created_by_user)) q = q.where(material_id: params[:material_id]) if params[:material_id].present? q = q.where(tag_name: params[:tag].to_s.strip) if params[:tag].present? q = q.where(event_type: params[:event_type]) if params[:event_type].present? @@ -20,7 +20,7 @@ class MaterialVersionsController < ApplicationController .limit(limit) .offset(offset) - render json: { versions: serialise_versions(versions), count: } + render json: { versions: serialise_versions(versions), count:, population_count: HistoryAccess.population_count(MaterialVersion) } end private diff --git a/backend/app/controllers/post_versions_controller.rb b/backend/app/controllers/post_versions_controller.rb index 2617cde..51f6f3f 100644 --- a/backend/app/controllers/post_versions_controller.rb +++ b/backend/app/controllers/post_versions_controller.rb @@ -14,21 +14,7 @@ class PostVersionsController < ApplicationController offset = (page - 1) * limit - q = PostVersion.joins(<<~SQL.squish) - LEFT JOIN - post_versions prev - ON - prev.post_id = post_versions.post_id - AND prev.version_no = post_versions.version_no - 1 - SQL - .select('post_versions.*', - 'prev.title AS prev_title', - 'prev.url AS prev_url', - 'prev.thumbnail_base AS prev_thumbnail_base', - 'prev.tags_json AS prev_tags_json', - 'prev.video_ms AS prev_video_ms', - 'prev.original_created_from AS prev_original_created_from', - 'prev.original_created_before AS prev_original_created_before') + q = HistoryAccess.visible(version_scope) q = q.where('post_versions.post_id = ?', post_id) if post_id if external_tag_id || (tag_id && !(Tag.exists?(id: tag_id))) q = q.where('JSON_CONTAINS(post_versions.tags_json,' + @@ -46,11 +32,23 @@ class PostVersionsController < ApplicationController .limit(limit) .offset(offset) - render json: { versions: serialise_versions(versions, locale.language_code), count: } + render json: { versions: serialise_versions(versions, locale.language_code), count:, population_count: HistoryAccess.population_count(PostVersion) } end private + def version_scope + PostVersion.with_previous_version + .select('post_versions.*', + 'prev.title AS prev_title', + 'prev.url AS prev_url', + 'prev.thumbnail_base AS prev_thumbnail_base', + 'prev.tags_json AS prev_tags_json', + 'prev.video_ms AS prev_video_ms', + 'prev.original_created_from AS prev_original_created_from', + 'prev.original_created_before AS prev_original_created_before') + end + def serialise_versions rows, language_code rows = rows.to_a user_ids = rows.map(&:created_by_user_id).compact.uniq @@ -103,7 +101,7 @@ class PostVersionsController < ApplicationController } missing_ids = ids - names.keys - NicoTagVersion + HistoryAccess.visible(NicoTagVersion.all) .where(tag_id: missing_ids) .order(:tag_id, version_no: :desc) .pluck(:tag_id, :name) diff --git a/backend/app/controllers/posts_controller.rb b/backend/app/controllers/posts_controller.rb index 9513e97..5665fa7 100644 --- a/backend/app/controllers/posts_controller.rb +++ b/backend/app/controllers/posts_controller.rb @@ -38,7 +38,7 @@ class PostsController < ApplicationController offset = (page - 1) * limit pt_max_sql = - PostVersion + HistoryAccess.visible(PostVersion.all) .select('post_id, MAX(created_at) AS max_updated_at') .group('post_id') .to_sql @@ -344,6 +344,7 @@ class PostsController < ApplicationController unless force base_version = post.post_versions.find_by!(version_no: base_version_no) + HistoryAccess.require!(base_version) base_snapshot = post_snapshot_from_version(base_version) current_snapshot = post_snapshot_from_record(post, locale) end @@ -408,68 +409,40 @@ class PostsController < ApplicationController target = Locale.find_by(code:, deprecated_at: nil) return head :not_found unless target - literals = params[:tags].to_s.split - references = tag_filter_references(literals) - names = literals.each_with_index.map do |literal, index| - reference = references&.[](index) - (reference && reference['negative'] == false) ? literal : literal.sub(/\Anot:/i, '') - end - internal = references ? {} : TagName.where( - language_code: source.language_code, name: names).pluck(:name, :tag_id).to_h - canonical = references ? {} : TagName.where( - language_code: source.language_code, primary_flg: true, - tag_id: internal.values).pluck(:tag_id, :name).to_h - external_names = names.map { canonical[internal[_1]] || _1 } - external = references ? {} : ExternalTag.where( - "CONCAT(platform, ':', name) IN (?)", external_names).pluck( - Arel.sql("CONCAT(platform, ':', name)"), :id).to_h - ids = references ? references.filter_map { _1['tag_id'] } : internal.values - translated = TagName.where(language_code: target.language_code, - primary_flg: true, tag_id: ids).pluck(:tag_id, :name).to_h + tokens = params[:tags].to_s.split + literals = PostTagSearch.literals(source, tokens.join(' ')) + source_ids = PostTagSearch.references_for(source, literals) + ids = source_ids.values.flat_map(&:first).uniq + names = TagName.where(language_code: target.language_code, + primary_flg: true, tag_id: ids).pluck(:tag_id, :name).to_h + filters = literals.map do |literal| + internal, = source_ids.fetch(literal[:name]) + name = internal.empty? ? literal[:name] : names[internal.first] + return head :unprocessable_entity unless name && internal.length <= 1 - filters = literals.each_with_index.map do |literal, index| - name = names[index] - reference = if references - references[index].except('name') - else - matches = {} - matches['tag_id'] = internal[name] if internal[name] - external_id = external[external_names[index]] - matches['external_tag_id'] = external_id if external_id - matches.empty? ? { 'missing' => true } : matches - end - name = translated[reference['tag_id']] || name - negative = reference.fetch('negative', literal.downcase.start_with?('not:')) - prefix = negative ? literal[/\Anot:/i] || 'not:' : '' - reference.merge('name' => "#{ prefix }#{ name }", 'negative' => negative) + literal[:negative] ? "not:#{ name }" : name end - render json: { references: filters } + translated = PostTagSearch.literals(target, filters.join(' ')) + target_ids = PostTagSearch.references_for(target, translated) + literals.zip(translated).each do |literal, translation| + unless (source_ids.fetch(literal[:name]).map(&:sort) == + target_ids.fetch(translation[:name]).map(&:sort)) + return head :unprocessable_entity + end + end + index = 0 + filters = tokens.map do |token| + next token if token.downcase == 'type:or' + + value = filters[index] + index += 1 + value + end + render json: { tags: filters.join(' ') } end private - def tag_filter_references literals - raw = params[:tag_refs] - return nil if raw.blank? - - references = JSON.parse(raw) - valid = references.is_a?(Array) && references.all? do |reference| - next false unless reference.is_a?(Hash) && reference['name'].is_a?(String) - - ids = [reference['tag_id'], reference['external_tag_id']].compact - ((ids.any? || reference['missing'] == true) && - ids.all? { _1.is_a?(Integer) && _1.positive? } && - (!reference.key?('missing') || (reference['missing'] == true && ids.empty?)) && - (!reference.key?('negative') || [true, false].include?(reference['negative']))) - end - raise ActionController::BadRequest unless valid - return nil unless references.map { _1['name'] } == literals - - references - rescue JSON::ParserError, TypeError - raise ActionController::BadRequest - end - def filtered_posts locale tag_names = params[:tags].to_s.split match_type = params[:match] @@ -481,29 +454,17 @@ class PostsController < ApplicationController end def filter_posts_by_tags locale, tag_names, match_type - references = tag_filter_references(tag_names) - literals = tag_names.each_with_index.map do |raw_name, index| - name = raw_name.sub(/\Anot:/i, '') - name = TagName.canonicalise(locale, name).first unless references - reference = references&.[](index) - negative = if reference&.key?('negative') - reference['negative'] - else - raw_name.downcase.start_with?('not:') - end - { name:, negative:, reference: } - end + literals = PostTagSearch.literals(locale, tag_names.join(' ')) return Post.all if literals.empty? if match_type == 'any' literals.reduce(Post.none) do |posts, literal| - posts.or(tag_literal_relation(locale, literal[:name], negative: literal[:negative], - reference: literal[:reference])) + posts.or(tag_literal_relation(locale, literal[:name], negative: literal[:negative])) end else literals.reduce(Post.all) do |posts, literal| - ids = tagged_post_ids_for(locale, literal[:name], reference: literal[:reference]) + ids = tagged_post_ids_for(locale, literal[:name]) if literal[:negative] posts.where.not(id: ids) else @@ -513,8 +474,8 @@ class PostsController < ApplicationController end end - def tag_literal_relation locale, name, negative:, reference: nil - ids = tagged_post_ids_for(locale, name, reference:) + def tag_literal_relation locale, name, negative: + ids = tagged_post_ids_for(locale, name) if negative Post.where.not(id: ids) else @@ -522,32 +483,11 @@ class PostsController < ApplicationController end end - def tagged_post_ids_for locale, name, reference: nil - if reference - matches = Post.none - if reference['tag_id'] - matches = matches.or(Post.where(id: PostTag.where( - tag_id: reference['tag_id']).select(:post_id))) - end - if reference['external_tag_id'] - matches = matches.or(Post.where(id: PostExternalTag.where( - external_tag_id: reference['external_tag_id']).select(:post_id))) - end - return matches.select(:id) - end - - posts_by_internal_tags = - Post - .joins(tags: :tag_names) - .where(tag_names: { language_code: locale.language_code, name: }) - .select(:id) - - posts_by_external_tags = - Post - .joins(:external_tags) - .where("CONCAT(external_tags.platform, ':', external_tags.name) = ?", name) - - (posts_by_internal_tags + posts_by_external_tags).uniq(&:id) + def tagged_post_ids_for locale, name + internal, external = PostTagSearch.references(locale, { name: }) + Post.where(id: PostTag.where(tag_id: Array(internal)).select(:post_id)) + .or(Post.where(id: PostExternalTag.where(external_tag_id: Array(external)) + .select(:post_id))).select(:id) end def sync_post_tags! post, desired_tags, sections diff --git a/backend/app/controllers/sitemap_entries_controller.rb b/backend/app/controllers/sitemap_entries_controller.rb index e681f81..14a85ff 100644 --- a/backend/app/controllers/sitemap_entries_controller.rb +++ b/backend/app/controllers/sitemap_entries_controller.rb @@ -42,7 +42,7 @@ class SitemapEntriesController < ApplicationController when 'descriptions' TagDescription.where(language_code: locale.language_code, tag_id: named_tags) .where.not(body: [nil, '']) - .where(TagDescriptionVersion + .where(HistoryAccess.visible(TagDescriptionVersion.all) .where('tag_description_versions.tag_id = tag_descriptions.tag_id') .where('tag_description_versions.language_code = tag_descriptions.language_code') .where('tag_description_versions.version_no = tag_descriptions.version_no') @@ -50,6 +50,7 @@ class SitemapEntriesController < ApplicationController when 'tag_groups' TagGroup.joins(:versions) .where('tag_group_versions.version_no = tag_groups.version_no') + .where(Arel.sql(HistoryAccess.sql(TagGroupVersion))) when 'theatres' Theatre.kept end diff --git a/backend/app/controllers/tag_descriptions_controller.rb b/backend/app/controllers/tag_descriptions_controller.rb index dfbaa4f..5c59226 100644 --- a/backend/app/controllers/tag_descriptions_controller.rb +++ b/backend/app/controllers/tag_descriptions_controller.rb @@ -49,6 +49,7 @@ class TagDescriptionsController < ApplicationController private def representation description + HistoryAccess.require_current!(description, TagDescriptionVersion) description.as_json(only: [ :tag_id, :language_code, diff --git a/backend/app/controllers/tag_groups_controller.rb b/backend/app/controllers/tag_groups_controller.rb index 1872f40..fc3ec26 100644 --- a/backend/app/controllers/tag_groups_controller.rb +++ b/backend/app/controllers/tag_groups_controller.rb @@ -1,7 +1,7 @@ class TagGroupsController < ApplicationController def index locale = resolve_locale! - groups = TagGroup.includes(:created_by_user, :updated_by_user).order(:id) + groups = HistoryAccess.current(TagGroup.all, TagGroupVersion).includes(:created_by_user, :updated_by_user).order(:id) render json: TagGroupRepr.many(locale, groups.to_a) end diff --git a/backend/app/controllers/tag_versions_controller.rb b/backend/app/controllers/tag_versions_controller.rb index e1169e9..95c71d0 100644 --- a/backend/app/controllers/tag_versions_controller.rb +++ b/backend/app/controllers/tag_versions_controller.rb @@ -10,17 +10,7 @@ class TagVersionsController < ApplicationController offset = (page - 1) * limit - q = TagVersion.joins(<<~SQL.squish) - LEFT JOIN - tag_versions prev - ON - prev.tag_id = tag_versions.tag_id - AND prev.version_no = tag_versions.version_no - 1 - SQL - .select('tag_versions.*', 'prev.name AS prev_name', 'prev.category AS prev_category', - 'prev.deprecated_at AS prev_deprecated_at', - 'prev.aliases AS prev_aliases', 'prev.names AS prev_names', - 'prev.parent_tag_ids AS prev_parent_tag_ids') + q = HistoryAccess.visible(version_scope) q = q.where('tag_versions.tag_id = ?', tag_id) if tag_id count = q.except(:select, :order, :limit, :offset).count @@ -29,11 +19,26 @@ class TagVersionsController < ApplicationController .limit(limit) .offset(offset) - render json: { versions: serialise_versions(versions, locale.language_code), count: } + render json: { versions: serialise_versions(versions, locale.language_code), count:, population_count: HistoryAccess.population_count(TagVersion) } end private + def version_scope + TagVersion.joins(<<~SQL.squish) + LEFT JOIN + tag_versions prev + ON + prev.tag_id = tag_versions.tag_id + AND prev.version_no = tag_versions.version_no - 1 + AND #{ HistoryAccess.sql(TagVersion, 'prev') } + SQL + .select('tag_versions.*', 'prev.name AS prev_name', 'prev.category AS prev_category', + 'prev.deprecated_at AS prev_deprecated_at', + 'prev.aliases AS prev_aliases', 'prev.names AS prev_names', + 'prev.parent_tag_ids AS prev_parent_tag_ids') + end + def serialise_versions rows, language_code rows = rows.to_a tag_ids = rows.map(&:tag_id).uniq diff --git a/backend/app/controllers/wiki_pages_controller.rb b/backend/app/controllers/wiki_pages_controller.rb index 966f511..9ac9992 100644 --- a/backend/app/controllers/wiki_pages_controller.rb +++ b/backend/app/controllers/wiki_pages_controller.rb @@ -8,12 +8,12 @@ class WikiPagesController < ApplicationController title = params[:title].to_s.strip descriptions = - TagDescription + HistoryAccess.current(TagDescription.all, TagDescriptionVersion) .where(language_code: locale.language_code) .includes(tag: :tag_names) descriptions = descriptions.order(updated_at: :desc, id: :asc) - tag_groups = TagGroup.order(updated_at: :desc, id: :asc) + tag_groups = HistoryAccess.current(TagGroup.all, TagGroupVersion).order(updated_at: :desc, id: :asc) if title.present? pattern = "%#{ ApplicationRecord.sanitize_sql_like(title) }%" descriptions = @@ -41,13 +41,13 @@ class WikiPagesController < ApplicationController return head :not_found unless language.is_a?(String) return head :not_found unless Language.exists?(code: language) - names = TagDescription.joins(tag: :tag_names) + names = HistoryAccess.current(TagDescription.all, TagDescriptionVersion).joins(tag: :tag_names) .where(language_code: locale.language_code, tag_names: { language_code: language }) .pluck('tag_descriptions.tag_id', 'tag_descriptions.id', 'tag_names.name') candidates = names.map do |tag_id, description_id, name| WikiPageRepr.autolink_tag(locale, tag_id:, description_id:, name:, language:) end - candidates.concat(TagGroup.order(:id).pluck(:id, :name).map do |id, name| + candidates.concat(HistoryAccess.current(TagGroup.all, TagGroupVersion).order(:id).pluck(:id, :name).map do |id, name| WikiPageRepr.autolink_group(locale, id:, name:, language:) end) render json: candidates @@ -286,12 +286,12 @@ class WikiPagesController < ApplicationController tag_group = id && TagGroup.find_by(id:) if tag_group - return render json: tag_group_changes(TagGroupVersion.where(tag_group_id: tag_group.id)) + return render json: tag_group_changes(HistoryAccess.visible(TagGroupVersion.all).where(tag_group_id: tag_group.id)) end json = [] unless id - json.concat(tag_group_changes(TagGroupVersion.all)) + json.concat(tag_group_changes(HistoryAccess.visible(TagGroupVersion.all))) json.concat(description_changes(locale, TagDescription.where(language_code: locale.language_code))) json = json.each_with_index.sort_by do |row, index| @@ -327,7 +327,7 @@ class WikiPagesController < ApplicationController def render_group_representation representation, version, body: nil, status: :ok body ||= TagGroupMarkdownRenderer.render(representation) - versions = TagGroupVersion.where(tag_group_id: version.tag_group_id) + versions = HistoryAccess.visible(TagGroupVersion.all).where(tag_group_id: version.tag_group_id) pred = versions.find_by(version_no: version.version_no - 1)&.id succ = versions.find_by(version_no: version.version_no + 1)&.id updated_at = version.created_at @@ -346,11 +346,11 @@ class WikiPagesController < ApplicationController end def current_group_version group - TagGroupVersion.find_by(tag_group_id: group.id, version_no: group.version_no) + HistoryAccess.visible(TagGroupVersion.all).find_by(tag_group_id: group.id, version_no: group.version_no) end def group_version group, version_id - TagGroupVersion.find_by(tag_group_id: group.id, id: version_id) + HistoryAccess.visible(TagGroupVersion.all).find_by(tag_group_id: group.id, id: version_id) end def wiki_diff from_body, to_body @@ -373,7 +373,7 @@ class WikiPagesController < ApplicationController def tag_group_changes versions rows = versions.includes(:created_by_user).order(id: :desc).limit(200).to_a groups = TagGroup.where(id: rows.map(&:tag_group_id).uniq).index_by(&:id) - predecessors = TagGroupVersion.where(tag_group_id: groups.keys, + predecessors = HistoryAccess.visible(TagGroupVersion.all).where(tag_group_id: groups.keys, version_no: rows.map { _1.version_no - 1 }) .pluck(:tag_group_id, :version_no, :id) .to_h { |group_id, number, version_id| [[group_id, number], version_id] } @@ -406,18 +406,18 @@ class WikiPagesController < ApplicationController end def description_versions description - TagDescriptionVersion.where(tag_id: description.tag_id, + HistoryAccess.visible(TagDescriptionVersion.all).where(tag_id: description.tag_id, language_code: description.language_code) end def tag_description_by_title locale, title - TagDescription.joins(tag: :tag_names) + HistoryAccess.current(TagDescription.all, TagDescriptionVersion).joins(tag: :tag_names) .find_by(language_code: locale.language_code, tag_names: { language_code: locale.language_code, name: title }) end def description_changes locale, descriptions - versions = TagDescriptionVersion.where(language_code: locale.language_code, + versions = HistoryAccess.visible(TagDescriptionVersion.all).where(language_code: locale.language_code, tag_id: descriptions.select(:tag_id)) .order(id: :desc).limit(200).to_a return [] if versions.empty? @@ -425,7 +425,7 @@ class WikiPagesController < ApplicationController by_tag = descriptions.where(tag_id: versions.map(&:tag_id)) .includes(tag: :tag_names).index_by(&:tag_id) users = User.where(id: versions.map(&:created_by_user_id)).index_by(&:id) - predecessors = TagDescriptionVersion.where( + predecessors = HistoryAccess.visible(TagDescriptionVersion.all).where( language_code: locale.language_code, tag_id: by_tag.keys, version_no: versions.map { |version| version.version_no - 1 }) .pluck(:tag_id, :version_no, :id) diff --git a/backend/app/models/history_access.rb b/backend/app/models/history_access.rb new file mode 100644 index 0000000..9e73210 --- /dev/null +++ b/backend/app/models/history_access.rb @@ -0,0 +1,214 @@ +require 'digest' + +# Request/worker authority comes only from a server-resolved user. Default is public. +class HistoryAccess < ActiveSupport::CurrentAttributes + attribute :user + class Forbidden < StandardError; end + ARTIFACT_POLICY = 'counts-masked-summary-1'.freeze + + KEYS = { 'post_versions' => ['post_id'], 'tag_versions' => ['tag_id'], + 'nico_tag_versions' => ['tag_id'], + 'tag_description_versions' => ['tag_id', 'language_code'], + 'tag_group_versions' => ['tag_group_id'], 'material_versions' => ['material_id'] }.freeze + + def self.revision + values = KEYS.keys.map do |table| + table.classify.constantize.unscoped.where(event_type: 'discard').maximum(:id) + end + Digest::SHA256.hexdigest(values.to_json) + end + + def self.authority + { 'artifact_policy' => ARTIFACT_POLICY, 'policy_revision' => revision, + 'admin' => admin?, 'actor_id' => user&.id } + end + + def self.authorised_artifact?(metadata) + return false unless metadata['artifact_policy'] == ARTIFACT_POLICY + return false unless metadata['policy_revision'].is_a?(String) + if metadata['admin'] + admin? && user.id == metadata['actor_id'] + else + metadata['policy_revision'] == revision + end + end + + def self.admin? = user&.admin? == true + + def self.sql(model, row = model.table_name, references: true) + return 'TRUE' if admin? + + table = model.table_name + keys = KEYS.fetch(table) + matches = keys.map { "boundary.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ') + boundaries = "SELECT #{ keys.join(', ') }, MAX(version_no) AS version_no FROM #{ table } " + + "WHERE event_type = 'discard' GROUP BY #{ keys.join(', ') }" + clauses = ["NOT EXISTS (SELECT 1 FROM (#{ boundaries }) boundary WHERE #{ matches } " + + "AND boundary.version_no >= #{ row }.version_no)"] + clauses += reference_sql(table, row) if references + clauses.map { "(#{ _1 })" }.join(' AND ') + end + + def self.discard_reference(table, key, expression) + "NOT EXISTS (SELECT 1 FROM (#{ discarded_keys(table, key) }) blocked " + + "WHERE #{ expression.call("blocked.#{ key }") })" + end + + def self.discarded_keys(table, key) + "SELECT #{ key } FROM #{ table } WHERE event_type = 'discard' GROUP BY #{ key }" + end + + def self.json_reference(table, key, document) + "NOT EXISTS (SELECT 1 FROM JSON_TABLE(COALESCE(#{ document }, JSON_ARRAY()), " + + "'$[*]' COLUMNS (reference_id BIGINT PATH '$' NULL ON EMPTY)) reference_entry " + + "JOIN (#{ discarded_keys(table, key) }) blocked " + + "ON reference_entry.reference_id = blocked.#{ key })" + end + + def self.list_reference(table, key, document) + # Preserve FIND_IN_SET's exact token semantics, including empty tokens. + array = "CONCAT('[', REPLACE(JSON_QUOTE(REPLACE(COALESCE(#{ document }, ''), " + + "' ', ',')), ',', '\",\"'), ']')" + "NOT EXISTS (SELECT 1 FROM JSON_TABLE(#{ array }, '$[*]' " + + "COLUMNS (reference_id VARCHAR(255) PATH '$')) reference_entry " + + "JOIN (#{ discarded_keys(table, key) }) blocked " + + "ON reference_entry.reference_id = CAST(blocked.#{ key } AS CHAR))" + end + + def self.snapshot_reference(model, row, external: false) + key = external ? 'external_tag_id' : 'tag_id' + table = model.table_name + source = "JSON_TABLE(COALESCE(#{ row }.tags_json, JSON_ARRAY()), '$[*]' COLUMNS (" + + "tag_id BIGINT PATH '$.#{ key }', version_no INT PATH '$.version_no')) tag_reference" + match = 'reference_version.tag_id = tag_reference.tag_id' + if external + match += " AND reference_version.created_at <= #{ row }.created_at" + latest = "AND NOT EXISTS (SELECT 1 FROM #{ table } later_reference " + + 'WHERE later_reference.tag_id = reference_version.tag_id ' + + "AND later_reference.created_at <= #{ row }.created_at " + + 'AND later_reference.version_no > reference_version.version_no)' + else + match += ' AND reference_version.version_no = tag_reference.version_no' + latest = '' + end + "NOT EXISTS (SELECT 1 FROM #{ source } JOIN #{ table } reference_version ON #{ match } " + + "WHERE NOT (#{ sql(model, 'reference_version') }) #{ latest })" + end + + def self.reference_sql(table, row) + case table + when 'post_versions' + [['tag_versions', 'tag_id', 'tag_id'], + ['nico_tag_versions', 'tag_id', 'external_tag_id']].map do |target, key, field| + json_reference(target, key, "JSON_EXTRACT(#{ row }.tags_json, '$[*].#{ field }')") + end + [list_reference('post_versions', 'post_id', "#{ row }.parent_post_ids"), + snapshot_reference(TagVersion, row), + snapshot_reference(NicoTagVersion, row, external: true)] + when 'nico_tag_versions' + # Legacy linked_tags snapshots contain names, not IDs. Match recorded names + # conservatively; never expose a private linked name through an external tag. + names = "JSON_MERGE_PRESERVE(JSON_ARRAY(private_tag.name), " + + "COALESCE(JSON_EXTRACT(private_tag.names, '$**.name'), JSON_ARRAY()))" + ["NOT EXISTS (SELECT 1 FROM tag_versions private_tag " + + "JOIN (#{ discarded_keys('tag_versions', 'tag_id') }) blocked " + + "ON blocked.tag_id = private_tag.tag_id " + + "JOIN JSON_TABLE(#{ names }, '$[*]' COLUMNS (name TEXT PATH '$')) private_name " + + "WHERE private_name.name <> '' AND " + + "LOCATE(CONCAT(' ', private_name.name, ' '), " + + "CONCAT(' ', COALESCE(#{ row }.linked_tags, ''), ' ')) > 0)"] + when 'tag_versions' + [list_reference('tag_versions', 'tag_id', "#{ row }.parent_tag_ids")] + when 'tag_description_versions' + [discard_reference('tag_versions', 'tag_id', ->(id) { "#{ row }.tag_id = #{ id }" })] + when 'material_versions' + [discard_reference('tag_versions', 'tag_id', ->(id) { "#{ row }.tag_id = #{ id }" }), + discard_reference('material_versions', 'material_id', ->(id) { + "#{ row }.parent_id = #{ id }" + })] + when 'tag_group_versions' + [['tag_versions', 'tag_id'], ['tag_group_versions', 'tag_group_id']].map do |target, field| + discard_reference(target, field, ->(id) { + ['members', 'placements'].map do |column| + "JSON_CONTAINS(COALESCE(JSON_EXTRACT(#{ row }.#{ column }, " + + "'$**.#{ field }'), JSON_ARRAY()), CAST(#{ id } AS CHAR))" + end.join(' OR ') + }) + end + else [] + end + end + + def self.visible(scope, row = scope.klass.table_name) + scope.where(Arel.sql(sql(scope.klass, row))) + end + + def self.current(scope, version_model) + return scope if admin? + + table = scope.klass.table_name + keys = KEYS.fetch(version_model.table_name) + matches = keys.map do |key| + column = scope.klass.column_names.include?(key) ? key : 'id' + "current_snapshot.#{ key } = #{ table }.#{ column }" + end.join(' AND ') + boundary = "#{ matches } AND current_snapshot.version_no >= #{ table }.version_no " + + "AND current_snapshot.event_type = 'discard'" + scope = scope.where(Arel.sql("NOT EXISTS (SELECT 1 FROM #{ version_model.table_name } " + + "current_snapshot WHERE #{ boundary })")) + condition = "#{ matches } AND current_snapshot.version_no = #{ table }.version_no" + scope.where(Arel.sql("NOT EXISTS (SELECT 1 FROM #{ version_model.table_name } " + + "current_snapshot WHERE #{ condition } AND NOT (#{ sql(version_model, 'current_snapshot') }))")) + end + + def self.require_current!(record, version_model) + return if admin? + raise Forbidden unless current(record.class.where(id: record.id), version_model).exists? + end + + def self.blob_visible?(blob, seen = []) + return true if admin? + return false if seen.include?(blob.id) + seen = seen + [blob.id] + private_versions = MaterialVersion.where(file_blob_id: blob.id) + .where(Arel.sql("NOT (#{ sql(MaterialVersion) })")) + return false if private_versions.exists? + + blob.attachments.each do |attachment| + model, key = { 'Post' => [PostVersion, 'post_id'], + 'Material' => [MaterialVersion, 'material_id'] }[attachment.record_type] + if model && model.where(key => attachment.record_id, event_type: 'discard').exists? + return false + end + if attachment.record_type == 'ActiveStorage::VariantRecord' + original = ActiveStorage::VariantRecord.find_by(id: attachment.record_id)&.blob + return false unless original && blob_visible?(original, seen) + end + end + true + end + + def self.require_many!(records) + return if admin? || records.empty? + + ids = records.map(&:id).uniq + raise Forbidden unless visible(records.first.class.where(id: ids)).count == ids.length + end + + def self.require!(record) + return record if admin? || visible(record.class.where(id: record.id)).exists? + + raise Forbidden + end + + # This number deliberately ignores time, identity and content filters to prevent count probes. + def self.population_count(model) = model.unscoped.count + + # Explicit whole-resource statistics; Counts pages do not run this aggregate. + def self.population_counts(model) + total, excluded = model.unscoped.optimizer_hints('MAX_EXECUTION_TIME(10000)').pick( + Arel.sql('COUNT(*)'), + Arel.sql("COALESCE(SUM(CASE WHEN #{ sql(model) } THEN 0 ELSE 1 END), 0)")) + { population_count: total.to_i, excluded_count: excluded.to_i, + excluded_count_scope: 'all_recorded_versions' } + end +end diff --git a/backend/app/models/post_version.rb b/backend/app/models/post_version.rb index 8fe52bb..4918c6e 100644 --- a/backend/app/models/post_version.rb +++ b/backend/app/models/post_version.rb @@ -1,6 +1,12 @@ class PostVersion < ApplicationRecord include VersionRecord + scope :with_previous_version, -> { + joins('LEFT JOIN post_versions prev ON prev.post_id = post_versions.post_id ' + + 'AND prev.version_no = post_versions.version_no - 1 AND ' + + HistoryAccess.sql(PostVersion, 'prev')) + } + belongs_to :post belongs_to :parent, class_name: 'Post', optional: true diff --git a/backend/app/models/tag_description_version.rb b/backend/app/models/tag_description_version.rb index 39f167d..cc8cd9a 100644 --- a/backend/app/models/tag_description_version.rb +++ b/backend/app/models/tag_description_version.rb @@ -1,3 +1,5 @@ class TagDescriptionVersion < ApplicationRecord + include VersionRecord + before_create { SharedResourceIdAllocator.assign!(self) } end diff --git a/backend/app/models/version_record.rb b/backend/app/models/version_record.rb index ccee900..15db840 100644 --- a/backend/app/models/version_record.rb +++ b/backend/app/models/version_record.rb @@ -6,6 +6,8 @@ module VersionRecord discard: 'discard', restore: 'restore' }.freeze + def require_history_access! = HistoryAccess.require!(self) + def readonly? = persisted? included do diff --git a/backend/app/representations/counts_version_repr.rb b/backend/app/representations/counts_version_repr.rb new file mode 100644 index 0000000..8073b7b --- /dev/null +++ b/backend/app/representations/counts_version_repr.rb @@ -0,0 +1,180 @@ +# Counts uses recorded values only; history/diff endpoints retain their own contract. +class CountsVersionRepr + # language_code remains a compatibility argument; it never selects recorded values. + def self.many(rows, resource:, language_code:, at: nil, inclusive: true, history_flg: false) + new(rows.to_a, resource:, language_code:, at:, inclusive:, history_flg:).serialise + end + + def initialize(rows, resource:, language_code:, at:, inclusive:, history_flg:) + @rows = rows + HistoryAccess.require_many!(rows) + @at = at + @inclusive = inclusive + @resource = resource + @history_flg = history_flg + @users = User.where(id: rows.map(&:created_by_user_id).compact).pluck(:id, :name).to_h + @languages = master_names(Language) + @scripts = master_names(Script) + @name_columns = CountsResourceDefinition.new('tags').name_columns + @tag_versions = {} + if resource == 'posts' + load_tag_versions + requests = @rows.flat_map do |row| + Array(json(row.tags_json)).filter_map do |entry| + [entry['external_tag_id'], parent_point(row)] if entry['external_tag_id'] + end + end.uniq + @external_versions = load_related_versions(NicoTagVersion, requests) + end + load_parent_versions if resource == 'tags' + end + + def serialise + @rows.map do |row| + resource_id = row.public_send(@resource == 'posts' ? :post_id : :tag_id) + common = { id: row.id, resource_id:, + version_no: row.version_no, latest: row.attributes['counts_latest'].to_i == 1, + created_at: row.created_at.iso8601, + created_by_user: row.created_by_user_id && + { id: row.created_by_user_id, name: @users[row.created_by_user_id] } } + common.merge(@resource == 'posts' ? post(row) : tag(row)) + end + end + + private + + def master_names(model) + model.select('code AS counts_code', :name).to_h do |record| + [record.attributes['counts_code'], record.name] + end + end + + def json(value) + value.is_a?(String) ? JSON.parse(value) : value + end + + def load_tag_versions + pairs = @rows.flat_map { Array(json(_1.tags_json)) } + .filter_map { |entry| [entry['tag_id'], entry['version_no']] if entry['tag_id'] } + .uniq + pairs.each_slice(500) do |batch| + placeholders = batch.map { '(?, ?)' }.join(', ') + scope = HistoryAccess.visible(TagVersion.all) + if @at + scope = scope.where("created_at #{ @inclusive ? '<=' : '<' } ?", @at) + end + scope.where(["(tag_id, version_no) IN (#{ placeholders })", *batch.flatten]) + .select(:tag_id, :version_no, :name, :category, :names, :created_at).each do |version| + @tag_versions[[version.tag_id, version.version_no]] = version + end + end + end + + def names(value) + (json(value) || {}).sort.map do |language, entry| + aliases = entry.fetch('aliases', []).map do |alias_entry| + alias_entry.merge('script_name' => @scripts[alias_entry['script_code']]) + end + { language_code: language, language_name: @languages[language], + script_code: entry['script_code'], script_name: @scripts[entry['script_code']], + name: entry['name'], auto_generated: entry['auto_generated'], aliases: } + end + end + + def locale_names(values) + @name_columns.to_h do |field| + locale = field.fetch(:name_locale) + if field[:multiple_values] + entries = values.select { _1[:language_code] == locale[:language_code] } + aliases = entries.flat_map do |entry| + entry[:aliases].map { _1.merge('language_code' => entry[:language_code]) } + end + next [field.fetch(:key).to_sym, aliases] + end + entry = values.find do |value| + value[:language_code] == locale[:language_code] + end + [field.fetch(:key).to_sym, entry && entry[:name]] + end + end + + def post(row) + snapshots = Array(json(row.tags_json)).map do |entry| + version = @tag_versions[[entry['tag_id'], entry['version_no']]] + version = nil if version && version.created_at > parent_point(row) + external = @external_versions[[entry['external_tag_id'], parent_point(row).iso8601(6)]] + recorded_names = version ? names(version.names) : [] + aliases = recorded_names.flat_map do |value| + value[:aliases].map { _1.merge('language_code' => value[:language_code], + 'language_name' => value[:language_name]) } + end + { id: entry['tag_id'] || entry['external_tag_id'], + kind: entry['tag_id'] ? 'internal' : 'external', + tag_id: entry['tag_id'], external_tag_id: entry['external_tag_id'], + version_no: entry['version_no'], + name: entry['name'] || version&.name || external&.fetch('name'), + category: entry['category'] || version&.category, + sections: entry.fetch('sections', []), names: recorded_names, aliases: } + .merge(locale_names(recorded_names)) + end + { title: row.title, url: row.url, thumbnail_base: row.thumbnail_base, + tags: snapshots, parents: row.parent_post_ids.to_s.split.map { { id: _1.to_i } }, + video_ms: row.video_ms, + original_created_from: row.original_created_from&.iso8601, + original_created_before: row.original_created_before&.iso8601 } + end + + def parent_point(row) + @history_flg ? [@at, row.created_at].compact.min : @at || row.created_at + end + + def load_parent_versions + requests = @rows.flat_map do |row| + row.parent_tag_ids.to_s.split.map { [_1.to_i, parent_point(row)] } + end.uniq + @parent_versions = load_related_versions(TagVersion, requests) + end + + def load_related_versions(model, requests) + result = {} + table = model.table_name + comparison = @inclusive ? '<=' : '<' + requests.each_slice(100) do |batch| + queries = batch.map do |id, point| + scope = HistoryAccess.visible(model.where(tag_id: id)) + .where("created_at #{ comparison } ?", point) + .where("NOT EXISTS (SELECT 1 FROM #{ table } newer " + + "WHERE newer.tag_id = #{ table }.tag_id AND newer.created_at #{ comparison } ? " + + "AND newer.version_no > #{ table }.version_no)", point) + quoted = model.connection.quote(point.iso8601(6)) + names = model == TagVersion ? 'names' : 'JSON_OBJECT() AS names' + scope.select("tag_id, name, #{ names }, #{ quoted } AS reference_point").to_sql + end + next if queries.empty? + + model.connection.select_all(queries.map { "(#{ _1 })" }.join(' UNION ALL ')).each do |value| + result[[value['tag_id'], value['reference_point']]] = value + end + end + result + end + + def tag(row) + recorded_names = names(row.names) + aliases = recorded_names.flat_map do |entry| + entry[:aliases].map { + _1.merge('language_code' => entry[:language_code], + 'language_name' => entry[:language_name]) + } + end + { name: row.name, names: recorded_names, + category: row.category, aliases:, + parents: row.parent_tag_ids.to_s.split.map do |id| + value = @parent_versions[[id.to_i, parent_point(row).iso8601(6)]] + snapshots = value && json(value['names']) + recorded = names(snapshots) + { id: id.to_i, name: value&.fetch('name'), names: recorded }.merge(locale_names(recorded)) + end, + deprecated: row.deprecated_at&.iso8601 }.merge(locale_names(recorded_names)) + end +end diff --git a/backend/app/representations/tag_group_repr.rb b/backend/app/representations/tag_group_repr.rb index 1b2afa4..94c9181 100644 --- a/backend/app/representations/tag_group_repr.rb +++ b/backend/app/representations/tag_group_repr.rb @@ -44,6 +44,7 @@ module TagGroupRepr end def one locale, group + HistoryAccess.require_current!(group, TagGroupVersion) many(locale, [group]).first end @@ -56,10 +57,11 @@ module TagGroupRepr end def current_version group - TagGroupVersion.find_by(tag_group_id: group.id, version_no: group.version_no) + HistoryAccess.visible(TagGroupVersion.all).find_by(tag_group_id: group.id, version_no: group.version_no) end def historical locale, version + HistoryAccess.require!(version) members = Array(version.members) placements = Array(version.placements) tag_references = members.filter_map { |member| member['tag_id'] } + @@ -153,13 +155,13 @@ module TagGroupRepr end def historical_tag_versions ids - TagVersion.where(tag_id: ids.uniq).to_a.index_by do |version| + HistoryAccess.visible(TagVersion.all).where(tag_id: ids.uniq).to_a.index_by do |version| [version.tag_id, version.version_no] end end def historical_group_versions ids - TagGroupVersion.where(tag_group_id: ids.uniq).to_a.index_by do |version| + HistoryAccess.visible(TagGroupVersion.all).where(tag_group_id: ids.uniq).to_a.index_by do |version| [version.tag_group_id, version.version_no] end end diff --git a/backend/app/representations/wiki_page_repr.rb b/backend/app/representations/wiki_page_repr.rb index 3a4aa21..5e8abb1 100644 --- a/backend/app/representations/wiki_page_repr.rb +++ b/backend/app/representations/wiki_page_repr.rb @@ -5,6 +5,7 @@ module WikiPageRepr module_function def tag_description locale, description + HistoryAccess.require_current!(description, TagDescriptionVersion) tag = description.tag { 'id' => description.id, diff --git a/backend/app/services/counts_axes.rb b/backend/app/services/counts_axes.rb new file mode 100644 index 0000000..3bb8dfb --- /dev/null +++ b/backend/app/services/counts_axes.rb @@ -0,0 +1,139 @@ +class CountsAxes + MAX_AXES = 8 + MASKED_VALUE = '__counts_private__'.freeze + PUBLIC_FIELDS = ['created_at', 'event_type'].freeze + class TooManyAxes < ArgumentError; end + class PrivateCondition < ArgumentError; end + + GRAINS = ['year', 'quarter', 'month', 'week', 'day', 'hour'].freeze + + def self.time_expression(expression, grain, timezone) + raise ArgumentError unless GRAINS.include?(grain) + + connection = ApplicationRecord.connection + # Use the installed TZInfo database, including DST, rather than MySQL timezone tables. + first = Time.utc(1900) + last = Time.utc(2101) + offset = timezone.period_for_utc(first).utc_total_offset + clauses = timezone.transitions_up_to(last, first).filter_map do |transition| + next if transition.offset.utc_total_offset == offset + + previous = offset + offset = transition.offset.utc_total_offset + "WHEN #{ expression } < #{ connection.quote(transition.at.to_time) } THEN #{ previous }" + end + seconds = clauses.empty? ? offset.to_s : "CASE #{ clauses.join(' ') } ELSE #{ offset } END" + local = "DATE_ADD(#{ expression }, INTERVAL (#{ seconds }) SECOND)" + case grain + when 'year' then "DATE_FORMAT(#{ local }, '%Y')" + when 'quarter' then "CONCAT(YEAR(#{ local }), '-Q', QUARTER(#{ local }))" + when 'month' then "DATE_FORMAT(#{ local }, '%Y-%m')" + when 'week' then "DATE_FORMAT(#{ local }, '%x-W%v')" + when 'day' then "DATE_FORMAT(#{ local }, '%Y-%m-%d')" + else "DATE_FORMAT(#{ local }, '%Y-%m-%d %H:00')" + end + end + + attr_reader :rows, :columns, :all + + def initialize(population) + @population = population + @definition = population.definition + @rows = parse(population.params.fetch('rows', '')) + @columns = parse(population.params.fetch('cols', '')) + @all = rows + columns + raise TooManyAxes if all.length > MAX_AXES + raise ArgumentError if all.map { _1['field'] }.uniq.length != all.length + end + + def expressions + @expressions ||= all.map do |axis| + field = axis.fetch('field') + expression = case field + when 'tags' + "CASE WHEN tag_entry.tag_id IS NOT NULL " + + "THEN CONCAT('internal:', tag_entry.tag_id) " + + "ELSE CONCAT('external:', tag_entry.external_tag_id) END" + when 'changed_field' then 'changed.field_key' + else + expression = @definition.expression(field) + @definition.fields[field] == :datetime ? self.class.time_expression( + expression, axis.fetch('grain', 'month'), @population.timezone) : expression + end + if !HistoryAccess.admin? && !PUBLIC_FIELDS.include?(field) + visible = HistoryAccess.sql(@definition.model, 'v') + if field == 'changed_field' + visible = "(#{ visible }) AND (#{ HistoryAccess.sql(@definition.model, 'previous') })" + end + expression = "CASE WHEN #{ visible } " + + "THEN CAST((#{ expression }) AS CHAR) ELSE '#{ MASKED_VALUE }' END" + end + expression + end + end + + def grouped_sql(scope, indexes = all.each_index.to_a) + keys = indexes.map { "a#{ _1 }" } + selected = indexes.map { "#{ expressions[_1] } AS a#{ _1 }" } + identities = ['v.id'] + if rows.any? { _1['field'] == 'tags' } + # Raw identities stay inside COUNT; masked tag occurrences remain additive. + identities << "COALESCE(CONCAT('internal:', tag_entry.tag_id), " + + "CONCAT('external:', tag_entry.external_tag_id))" + identities << 'changed.field_key' if rows.any? { _1['field'] == 'changed_field' } + end + members = identities.each_index.map { "counts_member_#{ _1 }" } + selected += identities.each_with_index.map { |expression, index| + "#{ expression } AS #{ members[index] }" + } + # Evaluate correlated visibility subqueries before grouping. LIMIT prevents + # derived-table merging, so ONLY_FULL_GROUP_BY sees ordinary projected columns. + projection = scope.except(:select, :order).select(Arel.sql(selected.join(', '))) + .limit(18_446_744_073_709_551_615).to_sql + fields = keys + ["COUNT(DISTINCT #{ members.join(', ') }) AS count"] + grouping = keys.empty? ? '' : " GROUP BY #{ keys.join(', ') }" + "SELECT #{ fields.join(', ') } FROM (#{ projection }) counts_values#{ grouping }" + end + + def joins(fields = all.map { _1['field'] }) + result = '' + if fields.include?('tags') + result += <<~SQL.squish + JOIN JSON_TABLE(CAST(v.tags_json AS JSON), '$[*]' COLUMNS ( + tag_id BIGINT PATH '$.tag_id' NULL ON EMPTY, + external_tag_id BIGINT PATH '$.external_tag_id' NULL ON EMPTY)) tag_entry + + SQL + end + if fields.include?('changed_field') + fields = @definition.changes.map do |field| + "SELECT #{ ApplicationRecord.connection.quote(field) } AS field_key" + end.join(' UNION ALL ') + condition = @definition.changes.map do |field| + "(changed.field_key = '#{ field }' AND #{ @population.changed_sql(field) })" + end.join(' OR ') + result += " JOIN (#{ fields }) changed ON (#{ condition })" + end + result + end + + private + + def parse(value) + axes = QueryState.axes(value) + raise ArgumentError unless axes.is_a?(Array) + + axes.each do |axis| + raise ArgumentError unless axis.is_a?(Hash) && (axis.keys - ['field', 'grain']).empty? + raise ArgumentError unless @definition.axes(@population.history).include?(axis['field']) + if @definition.fields[axis['field']] == :datetime + raise ArgumentError unless GRAINS.include?(axis.fetch('grain', 'month')) + + axis['grain'] ||= 'month' + elsif axis.key?('grain') + raise ArgumentError + end + end + axes + end +end diff --git a/backend/app/services/counts_candidates.rb b/backend/app/services/counts_candidates.rb new file mode 100644 index 0000000..ded9d1d --- /dev/null +++ b/backend/app/services/counts_candidates.rb @@ -0,0 +1,58 @@ +# Bounded, authorised candidates at the same recorded point as the Counts query. +class CountsCandidates + KEYS = ['resource', 'field', 'query', 'category', 'page', 'locale', 'at', + 'execution_at', 'timezone'].freeze + LIMIT = 20 + + def initialize(params) + QueryState.validate!(params, KEYS) + raise ArgumentError unless params.fetch('resource') == 'posts' + raise ArgumentError unless ['tags', 'tags_json'].include?(params.fetch('field')) + + @params = params + @page = Integer(params.fetch('page', '1'), 10) + raise ArgumentError unless @page.between?(1, 100_000) + raise ArgumentError if params.fetch('query', '').bytesize > 1000 + category = params['category'] + raise ArgumentError if category.present? && !Tag.categories.key?(category) + end + + def call + options = @params.slice('locale', 'at', 'execution_at', 'timezone') + .merge('definition' => 'latest', 'resource' => 'tags') + internal = CountsPopulation.new(options) + external = CountsPopulation.new(options.merge('resource' => 'nico_tags', + 'execution_at' => internal.at.iso8601(6))) + connection = TagVersion.connection + sources = [[internal, 'internal', 'v.category', 'v.names'], + [external, 'external', 'NULL', 'JSON_OBJECT()']] + sources = sources.map do |population, kind, category, names| + scope = population.scope.except(:select) + if @params['query'].present? + pattern = "%#{ ApplicationRecord.sanitize_sql_like(@params['query']) }%" + scope = scope.where("(v.name LIKE ? OR CAST(v.tag_id AS CHAR) = ? OR EXISTS (" + + "SELECT 1 FROM JSON_TABLE(COALESCE(JSON_EXTRACT(#{ names }, '$**.name'), " + + "JSON_ARRAY()), '$[*]' COLUMNS (name TEXT PATH '$')) candidate_name " + + 'WHERE candidate_name.name LIKE ?))', pattern, @params['query'], pattern) + end + scope.select(Arel.sql( + "CONCAT('#{ kind }:', v.tag_id) AS value, v.tag_id AS id, " + + "v.name AS name, #{ names } AS names, #{ category } AS category, '#{ kind }' AS kind")) + .to_sql + end + clauses = [] + if @params['category'].present? + clauses << ApplicationRecord.sanitize_sql_array(['category = ?', @params['category']]) + end + where = clauses.empty? ? '' : "WHERE #{ clauses.join(' AND ') }" + sql = <<~SQL + SELECT /*+ MAX_EXECUTION_TIME(10000) */ * FROM (#{ sources.join(' UNION ALL ') }) choices + #{ where } ORDER BY kind, id LIMIT #{ LIMIT + 1 } OFFSET #{ (@page - 1) * LIMIT } + SQL + # Sources already contain quoted search terms. Do not reinterpret their % or ? as binds. + rows = connection.select_all(sql).to_a + rows.each { _1['names'] = JSON.parse(_1['names']) if _1['names'].is_a?(String) } + { rows: rows.first(LIMIT), page: @page, more: rows.length > LIMIT, + execution_at: internal.at.iso8601(6) } + end +end diff --git a/backend/app/services/counts_condition.rb b/backend/app/services/counts_condition.rb new file mode 100644 index 0000000..1104c0f --- /dev/null +++ b/backend/app/services/counts_condition.rb @@ -0,0 +1,454 @@ +# Typed predicates over the same recorded fields used by Counts output. +# Multi-valued comparisons use EXISTS; inequality means no matching element. +class CountsCondition + CHILD_TYPES = { 'id' => :integer, 'versionNo' => :integer, 'tagId' => :integer, + 'tagGroupId' => :integer, 'autoGenerated' => :boolean, + 'category' => :enum, 'kind' => :enum, 'sections' => :interval }.freeze + + def self.fields(definition) + result = definition.metadata.map { _1.merge(labels: [definition.field_label(_1[:key])]) } + visit = lambda do |fields, prefix, labels| + fields.each do |field| + key = [prefix, field[:key]].compact.join('.') + path_labels = labels + [field[:label]] + type = if prefix + CHILD_TYPES.fetch(field[:key], :string) + elsif key == 'latest' + :boolean + elsif key == 'parents' + :integer + elsif key == 'deprecated' + :datetime + elsif key == 'thumbnail' + :string + else + definition.fields.fetch(key, :string) + end + type = :string if field[:children] || type == :json + type = :integer if ['parents', 'members'].include?(key) + physical = result.find { _1[:key] == key } + nullable = prefix.present? || (physical ? physical[:nullable] : key != 'latest') + nullable = false if field[:children] + entry = { key:, type:, labels: path_labels, nullable:, + multiple: prefix.present? || field[:children].present? || field[:multiple_values], + collection: field[:children].present? || type == :interval, + operators: CountsResourceDefinition::OPERATORS.fetch(type) } + entry[:label_values] = field[:label_values] if field[:label_values] + entry[:name_locale] = field[:name_locale] if field[:name_locale] + entry[:legacy] = true if field[:legacy] + result.reject! { _1[:key] == key } + result << entry + visit.call(field[:children] || [], key, path_labels) + end + end + visit.call(definition.condition_fields, nil, []) + # The raw JSON alias has the same membership semantics as the public tag field. + if definition.resource == 'posts' + result.find { _1[:key] == 'tags_json' }.merge!(type: :string, multiple: true, collection: true, + operators: CountsResourceDefinition::OPERATORS.fetch(:string)) + end + result += [{ key: 'user.id', type: :integer, nullable: true, + labels: ['history:actor', 'counts:columns.resourceId'], + operators: CountsResourceDefinition::OPERATORS.fetch(:integer) }, + { key: 'user.name', type: :string, nullable: true, + labels: ['history:actor', 'history:name'], + operators: CountsResourceDefinition::OPERATORS.fetch(:string) }] + result.each do |field| + next if field[:type] == :interval + + if field[:multiple] + field[:operators] = field[:operators] - ['in', 'not_in'] + ['any', 'all', 'none'] + end + field[:input_types] = ['single', 'multiple'] + if ['category', 'tag_category'].include?(field[:key].split('.').last) + field[:candidate_source] = 'categories' + field[:choices] = Tag.categories.keys.map { |value| + { value:, label: "common:categories.#{ value }" } + } + elsif definition.resource == 'posts' && ['tags', 'tags_json'].include?(field[:key]) + field[:candidate_source] = 'tags' + field[:set_type] = 'tag_reference' + end + field[:input_types] << 'list' if field[:candidate_source] + end + result + end + + def initialize(population) + @population = population + @definition = population.definition + @connection = @definition.model.connection + end + + def apply(scope, entry) + raise ArgumentError unless entry.is_a?(Hash) + if entry['operator'] == 'axis_eq' + raise ArgumentError unless (entry.keys - ['field', 'operator', 'grain', 'value']).empty? + value = entry['value'] + unless value.nil? || value.is_a?(String) || value.is_a?(Numeric) || [true, false].include?(value) + raise ArgumentError + end + raise ArgumentError if value.is_a?(String) && value.bytesize > 8192 + + return @population.axis_condition(scope, entry) + end + allowed = ['field', 'operator', 'value', 'end', 'blank', 'input', 'values'] + raise ArgumentError if (entry.keys - allowed).any? + + field = @definition.filter_metadata.find { _1[:key] == entry.fetch('field') } + raise ArgumentError unless field && field[:operators].include?(entry.fetch('operator')) + raise ArgumentError unless [true, false, nil].include?(entry['blank']) + input = entry.fetch('input', 'single') + raise ArgumentError unless ['single', 'multiple', 'list'].include?(input) + if input == 'list' && !field[:candidate_source] + raise ArgumentError + end + values = [entry['value'], entry['end']] + raise ArgumentError unless values.all? { _1.nil? || (_1.is_a?(String) && _1.bytesize <= 8192) } + if entry['operator'] != 'range' && !values[1].nil? && values[1] != '' + raise ArgumentError + end + + source, expression = source_for(field[:key]) + type = field[:type] + blank = blank_sql(expression, type) + if entry['blank'] + sql = if source && field[:collection] + "NOT EXISTS (SELECT 1 FROM #{ source })" + elsif source + "NOT EXISTS (SELECT 1 FROM #{ source } WHERE NOT (#{ blank }))" + else blank + end + return scope.where(Arel.sql(sql)) + end + if ['in', 'not_in', 'any', 'all', 'none'].include?(entry['operator']) + return set_condition(scope, entry, field) + end + raise ArgumentError if entry.key?('values') || input != 'single' + return scope if values.all? { _1.nil? || _1 == '' } + + parsed = values.map { parse(_1, type) } + validate_enum!(field[:key], parsed.first) if type == :enum + operator = entry['operator'] + if type == :interval + return interval(scope, source, expression, parsed) + end + clauses = [] + binds = [] + if operator == 'range' + ['>=', '<='].each_with_index do |comparison, index| + next if parsed[index].nil? + + clauses << "#{ expression } #{ comparison } ?" + binds << parsed[index] + end + raise ArgumentError if parsed.all? && parsed.first > parsed.last + elsif !parsed.first.nil? + comparison = { 'eq' => '=', 'ne' => '=', 'gt' => '>', 'lt' => '<', + 'ge' => '>=', 'le' => '<=', 'like' => 'LIKE' }.fetch(operator) + clauses << "#{ expression } #{ comparison } ?" + binds << parsed.first + end + return scope if clauses.empty? + + predicate = clauses.join(' AND ') + if source + predicate = "EXISTS (SELECT 1 FROM #{ source } WHERE #{ predicate })" + predicate = "NOT (#{ predicate })" if operator == 'ne' + elsif operator == 'ne' + predicate = "(NOT (#{ predicate }) OR #{ blank })" + end + scope.where(predicate, *binds) + end + + private + + def set_condition(scope, entry, field) + return scope unless entry.key?('values') + + values = entry['values'] + raise ArgumentError unless values.is_a?(Array) && values.length <= 100 + raise ArgumentError if [entry['value'], entry['end']].any? { !_1.nil? && _1 != '' } + + parsed = values.map do |value| + unless (value.is_a?(String) || value.is_a?(Numeric) || [true, false].include?(value)) + raise ArgumentError + end + raise ArgumentError if value.to_s.bytesize > 8192 + + if field[:set_type] == 'tag_reference' + raise ArgumentError unless value.is_a?(String) && + value.match?(/\A(?:internal|external):[1-9][0-9]{0,17}\z/) + value + else + raise ArgumentError if value == '' && ![:string, :enum].include?(field[:type]) + result = value == '' ? '' : parse(value.to_s, field[:type]) + validate_enum!(field[:key], result) if field[:type] == :enum + result + end + end + raise ArgumentError unless parsed.uniq == parsed + + operator = entry['operator'] + if parsed.empty? + return ['in', 'any'].include?(operator) ? scope.none : scope + end + source, expression = source_for(field[:key]) + if field[:set_type] == 'tag_reference' + source = post_tag_source + id = "COALESCE(#{ scalar('tag_snapshot.item', 'tag_id', :integer) }, " + + "#{ scalar('tag_snapshot.item', 'external_tag_id', :integer) })" + expression = "CONCAT(CASE WHEN #{ scalar('tag_snapshot.item', 'tag_id') } IS NULL " + + "THEN 'external:' ELSE 'internal:' END, #{ id })" + end + predicates = parsed.map do |_value| + clause = "#{ expression } = ?" + clause = "EXISTS (SELECT 1 FROM #{ source } WHERE #{ clause })" if source + clause + end + conjunction = operator == 'all' ? ' AND ' : ' OR ' + predicate = "(#{ predicates.join(conjunction) })" + if ['not_in', 'none'].include?(operator) + predicate = source ? "NOT #{ predicate }" : + "(NOT #{ predicate } OR #{ blank_sql(expression, field[:type]) })" + end + scope.where(predicate, *parsed) + end + + def parse(value, type) + return nil if value.nil? || value == '' + + case type + when :integer, :interval + raise ArgumentError unless value.match?(/\A-?[0-9]{1,19}\z/) + + number = Integer(value, 10) + raise ArgumentError unless number.between?(-9_223_372_036_854_775_808, 9_223_372_036_854_775_807) + + number + when :boolean + raise ArgumentError unless ['true', 'false'].include?(value) + + value == 'true' ? 1 : 0 + when :datetime then @population.timestamp(value) + else value + end + end + + def validate_enum!(key, value) + allowed = case key.split('.').last + when 'event_type' then VersionRecord::DEFAULT_EVENT_TYPE_MAP.values + when 'category', 'tag_category' then TagVersion.categories.keys + when 'kind' then ['internal', 'external'] + when 'source_kind' then Material::SOURCE_KINDS + end + raise ArgumentError if value && allowed && !allowed.include?(value) + end + + def blank_sql(expression, type) + if [:string, :enum].include?(type) + "(#{ expression } IS NULL OR #{ expression } = '')" + else + "#{ expression } IS NULL" + end + end + + def scalar(document, key, type = :string) + path = @connection.quote("$.#{ key }") + value = "JSON_EXTRACT(#{ document }, #{ path })" + text = "JSON_UNQUOTE(NULLIF(#{ value }, CAST('null' AS JSON)))" + return "CAST(#{ text } AS SIGNED)" if type == :integer + return "CASE #{ text } WHEN 'true' THEN 1 WHEN 'false' THEN 0 END" if type == :boolean + + text + end + + def json_table(document, path, name) + "JSON_TABLE(COALESCE(#{ document }, JSON_ARRAY()), #{ @connection.quote(path) } " + + "COLUMNS (item JSON PATH '$')) #{ name }" + end + + def names_source(document, alias_name) + source = "JSON_TABLE(COALESCE(JSON_KEYS(#{ document }), JSON_ARRAY()), '$[*]' " + + "COLUMNS (language_code TEXT PATH '$')) #{ alias_name }" + path = "CONCAT('$.', JSON_QUOTE(#{ alias_name }.language_code))" + [source, "JSON_EXTRACT(#{ document }, #{ path })", "#{ alias_name }.language_code"] + end + + def point + @point ||= @population.history ? @population.at : + @population.timestamp(@population.params['at']) || @population.at + end + + def name_source(row, name) + document = "JSON_MERGE_PRESERVE(JSON_ARRAY(#{ row }.name), " + + "COALESCE(JSON_EXTRACT(#{ row }.names, '$.*.name'), JSON_ARRAY()))" + [json_table(document, '$[*]', name), "JSON_UNQUOTE(#{ name }.item)"] + end + + def parent_source + column = @definition.resource == 'posts' ? 'parent_post_ids' : 'parent_tag_ids' + ids = "TRIM(REGEXP_REPLACE(COALESCE(v.#{ column }, ''), '[[:space:]]+', ' '))" + document = "CONCAT('[', REPLACE(#{ ids }, ' ', ','), ']')" + source = json_table(document, '$[*]', 'parent_entry') + id = 'CAST(parent_entry.item AS UNSIGNED)' + return [source, id] if @definition.resource == 'posts' + + bound = @population.history ? 'v.created_at' : @connection.quote(point) + source += ' LEFT JOIN tag_versions parent_version ON parent_version.tag_id = ' + id + + " AND parent_version.created_at <= #{ bound } AND " + + @population.access_sql(TagVersion, 'parent_version') + + ' AND NOT EXISTS (SELECT 1 FROM tag_versions newer_parent ' + + 'WHERE newer_parent.tag_id = parent_version.tag_id ' + + "AND newer_parent.created_at <= #{ bound } " + + 'AND newer_parent.version_no > parent_version.version_no)' + [source, id] + end + + def post_tag_source + source = json_table('v.tags_json', '$[*]', 'tag_snapshot') + source += ' LEFT JOIN tag_versions referenced_tag ON referenced_tag.tag_id = ' + + scalar('tag_snapshot.item', 'tag_id', :integer) + + ' AND referenced_tag.version_no = ' + scalar('tag_snapshot.item', 'version_no', :integer) + + " AND referenced_tag.created_at <= #{ @population.history ? 'v.created_at' : + @connection.quote(point) } AND " + + @population.access_sql(TagVersion, 'referenced_tag') + source + end + + def language_field(source, document, parts, inherited_language = nil) + return [source, scalar(document, 'name')] if parts.empty? + return [source, inherited_language] if parts == ['languageCode'] && inherited_language + + if parts.first == 'aliases' + aliases = "JSON_EXTRACT(#{ document }, '$.aliases')" + source += ' JOIN ' + json_table(aliases, '$[*]', 'alias_entry') + child = parts[1] || 'name' + return [source, inherited_language] if child == 'languageCode' && inherited_language + columns = { 'scriptCode' => 'script_code', 'name' => 'name' } + key = columns.fetch(child) { raise ArgumentError } + return [source, scalar('alias_entry.item', key)] + end + key = { 'name' => 'name', 'scriptCode' => 'script_code', + 'autoGenerated' => 'auto_generated' }.fetch(parts.first) { raise ArgumentError } + [source, scalar(document, key, CHILD_TYPES.fetch(parts.first, :string))] + end + + def source_for(key) + root, *parts = key.split('.') + if @definition.resource == 'tags' && @definition.name_column(root) + return @definition.alias_source(root) if root.start_with?('aliases_') + return [nil, @definition.name_expression(root)] + end + return [nil, "(#{ @population.current_latest_sql })"] if root == 'latest' + if root == 'user' && parts.any? + expression = parts.first == 'id' ? 'v.created_by_user_id' : + '(SELECT actor.name FROM users actor WHERE actor.id = v.created_by_user_id)' + return [nil, expression] + end + return [nil, 'v.thumbnail_base'] if root == 'thumbnail' + return [nil, 'v.deprecated_at'] if root == 'deprecated' + if root == 'parents' + source, id = parent_source + if @definition.resource == 'tags' && @definition.name_column(parts.first) + if parts.first.start_with?('aliases_') + aliases, value = @definition.alias_source(parts.first, 'parent_version') + return ["#{ source } JOIN #{ aliases }", value] + end + return [source, @definition.name_expression(parts.first, 'parent_version')] + end + if parts.first == 'names' + names, document, language = names_source('parent_version.names', 'parent_language') + return language_field("#{ source } JOIN #{ names }", document, parts.drop(1), language) + end + return [source, id] unless parts.first == 'name' + + names, value = name_source('parent_version', 'parent_name') + return ["#{ source } JOIN #{ names }", value] + end + if root == 'tags' || root == 'tags_json' + return post_field(parts) + end + if root == 'names' || root == 'aliases' + source, document, language = names_source('v.names', 'name_language') + parts = ['aliases', *parts] if root == 'aliases' + return language_field(source, document, parts, language) + end + if root == 'members' || root == 'placements' + path = '$[*]' + parts.take_while { _1 == 'children' }.map { '.children[*]' }.join + source = json_table("v.#{ root }", path, 'structure_entry') + if root == 'members' && parts.empty? + value = "COALESCE(#{ scalar('structure_entry.item', 'tag_id', :integer) }, " + + "#{ scalar('structure_entry.item', 'tag_group_id', :integer) })" + return [source, value] + end + leaf = parts.last || 'sectionName' + leaf = 'sectionName' if leaf == 'children' + column = leaf.gsub(/[A-Z]/) { "_#{ _1.downcase }" } + return [source, scalar('structure_entry.item', column, CHILD_TYPES.fetch(leaf, :string))] + end + if root == 'export_paths_json' + source, value, profile = names_source('v.export_paths_json', 'export_profile') + return [source, parts.first == 'profile' ? profile : "JSON_UNQUOTE(#{ value })"] + end + return name_source('v', 'recorded_name') if root == 'name' && @definition.resource == 'tags' + + [nil, @definition.expression(root)] + end + + def post_field(parts) + source = post_tag_source + document = 'tag_snapshot.item' + if @definition.name_column(parts.first) + if parts.first.start_with?('aliases_') + aliases, value = @definition.alias_source(parts.first, 'referenced_tag') + return ["#{ source } JOIN #{ aliases }", value] + end + return [source, @definition.name_expression(parts.first, 'referenced_tag')] + end + case parts.first + when 'names', 'aliases' + names, entry, language = names_source('referenced_tag.names', 'tag_language') + fields = parts.first == 'aliases' ? ['aliases', *parts.drop(1)] : parts.drop(1) + language_field("#{ source } JOIN #{ names }", entry, fields, language) + when 'sections' + sections = "JSON_EXTRACT(#{ document }, '$.sections')" + source += ' JOIN ' + json_table(sections, '$[*]', 'section_entry') + [source, 'section_entry.item'] + when 'id' + [source, "COALESCE(#{ scalar(document, 'tag_id', :integer) }, " + + "#{ scalar(document, 'external_tag_id', :integer) })"] + when 'versionNo' then [source, scalar(document, 'version_no', :integer)] + when 'kind' + [source, "CASE WHEN #{ scalar(document, 'tag_id') } IS NULL " + + "THEN 'external' ELSE 'internal' END"] + when 'category' + [source, "COALESCE(#{ scalar(document, 'category') }, referenced_tag.category)"] + else + names = "JSON_MERGE_PRESERVE(JSON_ARRAY(#{ scalar(document, 'name') }, " + + 'referenced_tag.name), ' + + "COALESCE(JSON_EXTRACT(referenced_tag.names, '$.*.name'), JSON_ARRAY()))" + ["#{ source } JOIN #{ json_table(names, '$[*]', 'recorded_name') }", + 'JSON_UNQUOTE(recorded_name.item)'] + end + end + + def interval(scope, source, document, bounds) + raise ArgumentError if bounds.compact.any?(&:negative?) + raise ArgumentError if bounds.all? && bounds.first > bounds.last + + first = scalar(document, 'begin_ms', :integer) + last = scalar(document, 'end_ms', :integer) + clauses = [] + values = [] + if bounds.first + clauses << "(#{ last } IS NULL OR #{ last } > ?)" + values << bounds.first + end + if bounds.last + clauses << "#{ first } < ?" + values << bounds.last + end + scope.where("EXISTS (SELECT 1 FROM #{ source } WHERE #{ clauses.join(' AND ') })", *values) + end +end diff --git a/backend/app/services/counts_defaults.rb b/backend/app/services/counts_defaults.rb new file mode 100644 index 0000000..8c41628 --- /dev/null +++ b/backend/app/services/counts_defaults.rb @@ -0,0 +1,94 @@ +# Only confirmed combinations belong here; all others retain the existing behaviour. +class CountsDefaults + POST_COLUMNS = ['resource_id', 'version_no', 'latest', 'title', 'url', + 'thumbnail_base', 'tags', 'parents', 'original_created_from', + 'original_created_before', 'event_type', 'user', 'created_at'].freeze + + def self.entries(resource, locale: 'ja') + result = [{ definition: 'history', mode: 'summary', params: { + 'rows' => 'created_at:month', + 'cols' => 'event_type', + 'sort' => 'a0:desc', 'conditions' => '' } }] + result << { definition: 'latest', mode: 'summary', params: { + 'rows' => '', 'cols' => '', 'conditions' => '', 'tags' => '' } } + details = detail_params(resource) + if details + details = details.merge('sort' => 'resource_id:asc version_no:asc') + language = Locale.find_by(code: locale)&.language_code || locale + name = CountsResourceDefinition.name_key(language) + aliases = CountsResourceDefinition.aliases_key(language) + details = details.transform_values do |value| + value.split(' ').flat_map do |token| + if resource == 'tags' && ['name', 'parents.name'].include?(token) + token == 'name' ? name : "parents.#{ name }" + elsif resource == 'tags' && token == 'aliases' + aliases + elsif resource == 'tags' && token.start_with?('aliases.') + [] + elsif resource == 'posts' && token == 'tags.name' + "tags.#{ name }" + else token + end + end.join(' ') + end + if ['posts', 'tags'].include?(resource) + fields = CountsResourceDefinition.new(resource).output_fields + selected = details.fetch('columns').split(' ') + details['columns'] = fields.map { _1[:key] }.select { selected.include?(_1) }.join(' ') + order = lambda do |items, prefix| + items.flat_map do |field| + path = [prefix, field[:key]].compact.join('.') + [path, *order.call(field.fetch(:children, []), path)] + end + end + details['order'] = order.call(fields, nil).join(' ') + end + ['latest', 'history'].each do |definition| + result << { definition:, mode: 'details', params: details } + end + end + result + end + + def self.detail_params(resource) + if resource == 'posts' + return { 'columns' => POST_COLUMNS.join(' '), 'order' => POST_COLUMNS.join(' '), + 'children' => 'tags.name parents.id', 'nested_json' => '0', + 'conditions' => '', 'tags' => '' } + end + unless resource == 'tags' + return { 'columns' => CountsExportProjection.default_columns(resource).join(' '), + 'children' => CountsExportProjection.default_children(resource).join(' ') } + end + + columns = ['resource_id', 'version_no', 'latest', 'name', 'category', + 'aliases', 'parents', 'deprecated', 'event_type', 'user', 'created_at'] + { 'columns' => columns.join(' '), 'order' => columns.join(' '), + 'children' => 'aliases.name parents.name', 'nested_json' => '0' } + end + + def self.for(params) + defaults = entries(params.fetch('resource', 'posts'), locale: params.fetch('locale', 'ja')) + settings = defaults.find do |entry| + entry[:definition] == params.fetch('definition', 'latest') && + entry[:mode] == params.fetch('mode', 'summary') + end&.fetch(:params) || {} + if settings.key?('sort') && params.key?('rows') && params.fetch('mode', 'summary') == 'summary' + actual = QueryState.axes(params['rows']) + expected = QueryState.axes(settings.fetch('rows')) + compatible = actual.is_a?(Array) && actual.first.is_a?(Hash) && + actual.first['field'] == expected.first['field'] && + actual.first.fetch('grain', 'month') == expected.first['grain'] + unless compatible + settings = settings.except('sort') + end + end + settings + end + + def self.metadata(resource, locale: 'ja') + entries(resource, locale:).map do |entry| + entry.merge(params: entry[:params].map { |key, value| { key:, value: } }) + end + end +end diff --git a/backend/app/services/counts_export_download.rb b/backend/app/services/counts_export_download.rb new file mode 100644 index 0000000..96c1d24 --- /dev/null +++ b/backend/app/services/counts_export_download.rb @@ -0,0 +1,154 @@ +require 'tempfile' +require 'fileutils' + +# Finish generation before reporting readiness; downloads never hold the full file in RAM. +class CountsExportDownload + EXPIRY = 10.minutes + MAX_CONCURRENT_EXPORTS = 2 + MAX_FILE_BYTES = 128.megabytes + MAX_STORED_BYTES = 512.megabytes + MAX_EXPORT_SECONDS = 120 + class Busy < StandardError; end + class TooLarge < StandardError; end + class TimedOut < StandardError; end + class StorageFull < StandardError; end + + # Rack body deliberately has no to_path: sendfile middleware must not bypass ACK. + class ResponseBody + def initialize body, &state + @body = body + @closed = false + @state = state + end + + def each &block + return enum_for(:each) unless block + return if @closed + + transmit(&block) + end + + def close + return if @closed + + @closed = true + @body.close if @body.respond_to?(:close) + end + + private + + def transmit + first = true + @body.each do |chunk| + raise IOError, 'Download no longer authorised' unless @state.call('active') + + yield chunk + if first + # Rack accepted a chunk; browser receipt and saving are not guaranteed. + raise IOError, 'Download acknowledgement failed' unless @state.call('started') + + first = false + end + end + @state.call('failed') if first + rescue StandardError => error + Rails.logger.warn("counts_download_stream_failure: #{ error.class }") + @state.call('failed') + raise + ensure + close + end + end + + def self.directory + Rails.root.join(ENV.fetch('COUNTS_EXPORT_DIRECTORY', 'tmp/counts_exports')) + end + def self.verifier = Rails.application.message_verifier('counts_export_download') + + def self.prepare(exporter, owner, progress: nil, phase: nil, check: nil, identifier: nil) + FileUtils.mkdir_p(directory, mode: 0o700) + authority = HistoryAccess.authority + slot = acquire_slot + + begin + generate(exporter, owner, progress:, phase:, check:, identifier:, authority:) + ensure + slot.close + end + end + + def self.acquire_slot + MAX_CONCURRENT_EXPORTS.times do |index| + path = directory.join(".slot-#{ index }.lock") + handle = File.open(path, File::RDWR | File::CREAT, 0o600) + return handle if handle.flock(File::LOCK_EX | File::LOCK_NB) + + handle.close + end + raise Busy + end + + def self.generate(exporter, owner, progress: nil, phase: nil, check: nil, + identifier: nil, authority: HistoryAccess.authority) + FileUtils.mkdir_p(directory, mode: 0o700) + Dir.glob(directory.join('counts-*.export')).each do |path| + File.unlink(path) if File.mtime(path) < Time.current - EXPIRY + rescue Errno::ENOENT + # Another request may have removed an expired download. + end + stored = Dir.glob(directory.join('counts-*')).sum do |path| + File.size(path) + rescue Errno::ENOENT + 0 + end + reserved = MAX_CONCURRENT_EXPORTS * MAX_FILE_BYTES + raise StorageFull if stored + reserved > MAX_STORED_BYTES + + prefix = identifier ? "counts-#{ identifier }-" : 'counts-' + file = Tempfile.create([prefix, '.part'], directory) + complete = false + bytes = 0 + deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + MAX_EXPORT_SECONDS + path = file.path + begin + file.binmode + exporter.body(progress:, phase:).each do |chunk| + check&.call + raise TimedOut if Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline + + bytes += chunk.bytesize + raise TooLarge if bytes > MAX_FILE_BYTES + + file.write(chunk) + end + file.flush + file.fsync + file.close + finished_path = path.sub(/\.part\z/, '.export') + File.rename(path, finished_path) + path = finished_path + token = verifier.generate({ 'file' => File.basename(path), + 'resource' => exporter.resource, 'format' => exporter.format, + 'owner' => owner, **authority }, expires_in: EXPIRY, purpose: 'counts_export') + complete = true + token + ensure + file.close unless file.closed? + File.unlink(path) if !complete && File.exist?(path) + end + end + + def self.find(token, owner) + metadata = verifier.verified(token, purpose: 'counts_export') + return unless metadata.is_a?(Hash) && metadata['owner'] == owner + return unless HistoryAccess.authorised_artifact?(metadata) + return unless metadata['file']&.match?(/\Acounts-[a-zA-Z0-9_-]+\.export\z/) + return unless CountsResourceDefinition::DEFINITIONS.key?(metadata['resource']) + return unless ['json', 'csv'].include?(metadata['format']) + + path = directory.join(metadata['file']) + return unless path.file? + + metadata.merge('path' => path.to_s) + end +end diff --git a/backend/app/services/counts_export_jobs.rb b/backend/app/services/counts_export_jobs.rb new file mode 100644 index 0000000..9e2cc1b --- /dev/null +++ b/backend/app/services/counts_export_jobs.rb @@ -0,0 +1,470 @@ +require 'json' +require 'fileutils' + +# A durable, bounded export spool. Web and supervised workers share the same POSIX volume. +class CountsExportJobs + class Cancelled < StandardError; end + class WorkerUnavailable < StandardError; end + ACTIVE = ['queued', 'running', 'cancelling'].freeze + HEARTBEAT_SECONDS = 15 + QUEUE_SECONDS = CountsExportDownload::MAX_EXPORT_SECONDS + 30 + + def self.worker_id + @worker_id ||= ".worker-#{ Process.pid }-#{ SecureRandom.hex(8) }" + end + + def self.pulse + FileUtils.mkdir_p(directory, mode: 0o700) + write({ 'id' => worker_id, 'heartbeat_at' => Time.current.to_f }) + end + + def self.worker_available? + Dir.glob(directory.join('.worker-*.json')).any? do |path| + JSON.parse(File.read(path)).fetch('heartbeat_at') > Time.current.to_f - HEARTBEAT_SECONDS + rescue Errno::ENOENT + false + end + end + + def self.expire_stalled(state) + now = Time.current.to_f + queued = state['status'] == 'queued' + stale = if queued + now - state.fetch('created_at', now) > QUEUE_SECONDS || + (!worker_available? && now - state.fetch('created_at', now) > HEARTBEAT_SECONDS) + else + ['running', 'cancelling'].include?(state['status']) && + now - state.fetch('heartbeat_at', now) > HEARTBEAT_SECONDS + end + return unless stale + + state.merge!('status' => 'failed', 'error' => queued ? 'worker_unavailable' : 'worker_stalled', + 'updated_at' => now) + remove_download(state) + state.delete('token') + write(state) + end + + def self.failure_code(error) + case error + when Cancelled then 'cancelled' + when CountsAxes::TooManyAxes then 'too_many_axes' + when CountsAxes::PrivateCondition then 'private_condition' + when CountsTableQuery::TooManyColumns then 'too_many_columns' + when ActiveRecord::StatementTimeout then 'counts_query_timeout' + when CountsExportDownload::TimedOut then 'generation_timeout' + when CountsExportDownload::TooLarge then 'export_too_large' + when CountsExportDownload::StorageFull, Errno::ENOSPC then 'export_storage_full' + else 'generation_failed' + end + end + + def self.directory = CountsExportDownload.directory.join('jobs') + def self.verifier = Rails.application.message_verifier('counts_export_job') + + def self.synchronise + FileUtils.mkdir_p(directory, mode: 0o700) + File.open(directory.join('.registry.lock'), File::RDWR | File::CREAT, 0o600) do |lock| + lock.flock(File::LOCK_EX) + yield + end + end + + def self.write(state) + path = directory.join("#{ state.fetch('id') }.json") + temporary = "#{ path }.#{ SecureRandom.hex(8) }.tmp" + File.open(temporary, 'w', 0o600) do |file| + file.write(JSON.generate(state)) + file.flush + file.fsync + end + File.rename(temporary, path) + ensure + File.unlink(temporary) if temporary && File.exist?(temporary) + end + + def self.read(id) + JSON.parse(File.read(directory.join("#{ id }.json"))) + rescue Errno::ENOENT + nil + end + + def self.remove_download(state) + name = state['download_file'] + return unless name&.match?(/\Acounts-[a-zA-Z0-9_-]+\.export\z/) + + File.unlink(CountsExportDownload.directory.join(name)) + rescue Errno::ENOENT + nil + end + + def self.remove_partial(id) + Dir.glob(CountsExportDownload.directory.join("counts-#{ id }-*")).each do |path| + File.unlink(path) + rescue Errno::ENOENT + nil + end + end + + def self.clean + Dir.glob(directory.join('*.tmp')).each do |path| + File.unlink(path) if File.mtime(path) < Time.current - CountsExportDownload::EXPIRY + end + Dir.glob(directory.join('*.run')).each do |path| + id = File.basename(path, '.run') + next if directory.join("#{ id }.json").exist? + + File.open(path, File::RDWR) do |lock| + File.unlink(path) if lock.flock(File::LOCK_EX | File::LOCK_NB) + end + end + Dir.glob(directory.join('*.json')).each do |path| + state = JSON.parse(File.read(path)) + next if state.fetch('expires_at') > Time.current.to_f + + run = directory.join("#{ state.fetch('id') }.run") + File.open(run, File::RDWR | File::CREAT, 0o600) do |lock| + unless lock.flock(File::LOCK_EX | File::LOCK_NB) + state['status'] = 'cancelling' + write(state) + next + end + remove_download(state) + remove_partial(state.fetch('id')) + File.unlink(path) + File.unlink(run) + end + end + end + + def self.start(payload, owner, language) + payload = payload.dup + time_key = payload.key?('definition') ? 'execution_at' : 'at' + if payload[time_key].nil? || payload[time_key] == '' + payload[time_key] = Time.current.utc.iso8601(6) + end + exporter = CountsExporter.new(payload, language_code: language) + authority = HistoryAccess.authority + synchronise do + clean + raise WorkerUnavailable unless worker_available? + active = Dir.glob(directory.join('*.json')).count do |path| + existing = JSON.parse(File.read(path)) + expire_stalled(existing) + ACTIVE.include?(existing['status']) + end + raise CountsExportDownload::Busy if active >= CountsExportDownload::MAX_CONCURRENT_EXPORTS + + id = SecureRandom.hex(24) + state = { 'id' => id, 'owner' => owner, 'payload' => payload, + 'language' => language, 'resource' => exporter.resource, + 'format' => exporter.format, 'status' => 'queued', + 'phase' => 'queued', 'created_at' => Time.current.to_f, + 'updated_at' => Time.current.to_f, 'heartbeat_at' => Time.current.to_f, + 'processed' => 0, 'total' => nil, + **authority, + 'expires_at' => (Time.current + CountsExportDownload::EXPIRY).to_f } + write(state) + verifier.generate({ 'id' => id, 'owner' => owner }, + expires_in: CountsExportDownload::EXPIRY, purpose: 'counts_export_job') + end + end + + def self.authorised_id(token, owner) + value = verifier.verified(token, purpose: 'counts_export_job') + return unless value.is_a?(Hash) && value['owner'] == owner + return unless value['id']&.match?(/\A[a-f0-9]{48}\z/) + + value['id'] + end + + def self.status(token, owner, cancel: false) + id = authorised_id(token, owner) + return unless id + snapshot = read(id) + return unless snapshot && snapshot['owner'] == owner + return unless HistoryAccess.authorised_artifact?(snapshot) + + synchronise do + state = read(id) + return unless state && state['owner'] == owner && state['expires_at'] > Time.current.to_f + + expire_stalled(state) + if (state['status'] == 'ready' && + ['pending', 'streaming'].include?(state['download_status']) && + state.fetch('download_expires_at', 0) <= Time.current.to_f) + state['download_status'] = 'timeout' + write(state) + end + + if cancel && ['queued', 'running', 'ready'].include?(state['status']) + state['status'] = state['status'] == 'running' ? 'cancelling' : 'cancelled' + remove_download(state) + state.delete('token') + write(state) + end + result = state.slice('status', 'phase', 'processed', 'total', 'format', 'token', 'error', + 'updated_at', 'heartbeat_at', 'download_attempt', 'download_status') + result['max_axes'] = CountsAxes::MAX_AXES if state['error'] == 'too_many_axes' + result['max_columns'] = CountsTableQuery::MAX_COLUMNS if state['error'] == 'too_many_columns' + result + end + end + + def self.download_attempt(token, owner, file_token: nil, attempt: nil, status: nil) + id = authorised_id(token, owner) + return unless id + + synchronise do + state = read(id) + return unless (state && state['owner'] == owner && state['status'] == 'ready' && + state['expires_at'] > Time.current.to_f && HistoryAccess.authorised_artifact?(state)) + if attempt + return unless state['download_attempt'] == attempt && state['token'] == file_token + return unless state['download_status'] == 'started' || + state.fetch('download_expires_at', 0) > Time.current.to_f + + transitions = { 'streaming' => ['pending'], 'started' => ['streaming'], + 'failed' => ['pending', 'streaming', 'started'], + 'unavailable' => ['pending', 'streaming'] } + if status == 'active' + return state.slice('download_attempt', 'download_status', 'token') if + ['streaming', 'started'].include?(state['download_status']) + return + end + if status + return unless transitions.fetch(status, []).include?(state['download_status']) + + state['download_status'] = status + else + return unless state['download_status'] == 'pending' + end + else + # Repeated acceptance POSTs do not invalidate an in-flight GET or issue duplicates. + if (state['download_status'] == 'started' || + (['pending', 'streaming'].include?(state['download_status']) && + state.fetch('download_expires_at', 0) > Time.current.to_f)) + return state.slice('download_attempt', 'download_status', 'token') + end + state['download_attempt'] = SecureRandom.hex(24) + state['download_status'] = 'pending' + state['download_expires_at'] = Time.current.to_f + 45 + end + write(state) + state.slice('download_attempt', 'download_status', 'token') + end + end + + # GET already verified authority. During streaming, observe cancellation without + # repeating the six-table policy revision aggregate for every 64KiB chunk. + def self.stream_active?(token, owner, file_token:, attempt:) + id = authorised_id(token, owner) + state = id && read(id) + state && state['owner'] == owner && state['status'] == 'ready' && + state['expires_at'] > Time.current.to_f && state['token'] == file_token && + state['download_attempt'] == attempt && + (state['download_status'] == 'started' || + (state['download_status'] == 'streaming' && + state.fetch('download_expires_at', 0) > Time.current.to_f)) + end + + def self.checkpoint(id, processed = nil, total = nil, phase: nil) + synchronise do + state = read(id) + unless state && state['status'] == 'running' && state['expires_at'] > Time.current.to_f + raise Cancelled + end + if processed || phase + state.merge!('processed' => processed, 'total' => total) unless processed.nil? + state['phase'] = phase if phase + state['updated_at'] = Time.current.to_f + write(state) + end + end + end + + def self.work_one + pulse + FileUtils.mkdir_p(directory, mode: 0o700) + Dir.glob(directory.join('*.json')).sort.each do |path| + id = File.basename(path, '.json') + File.open(directory.join("#{ id }.run"), File::RDWR | File::CREAT, 0o600) do |lock| + next unless lock.flock(File::LOCK_EX | File::LOCK_NB) + + state = synchronise do + candidate = read(id) + next unless candidate && ACTIVE.include?(candidate['status']) + + if candidate['status'] != 'queued' + candidate['status'] = candidate['status'] == 'cancelling' ? 'cancelled' : 'failed' + remove_partial(id) + candidate['error'] = 'interrupted' + write(candidate) + next + end + candidate['status'] = 'running' + candidate['phase'] = 'aggregating' + candidate['heartbeat_at'] = Time.current.to_f + candidate['updated_at'] = Time.current.to_f + write(candidate) + candidate + end + next unless state + + supervise(state) + return true + end + end + synchronise { clean } + false + end + + def self.supervise(state) + id = state.fetch('id') + ApplicationRecord.connection_handler.clear_all_connections! + pid = Process.fork do + Signal.trap('TERM') { raise Cancelled } + Signal.trap('INT') { raise Cancelled } + begin + perform(state) + ensure + ApplicationRecord.connection_handler.clear_all_connections! + Process.exit!(0) + end + end + started = Process.clock_gettime(Process::CLOCK_MONOTONIC) + termination = nil + reaped = false + begin + loop do + if Process.waitpid(pid, Process::WNOHANG) + reaped = true + break + end + pulse + now = Process.clock_gettime(Process::CLOCK_MONOTONIC) + terminal = synchronise do + current = read(id) + if current && current['status'] == 'running' + if now - started > CountsExportDownload::MAX_EXPORT_SECONDS + current.merge!('status' => 'failed', 'error' => 'generation_timeout') + end + current['heartbeat_at'] = Time.current.to_f + write(current) + end + !current || !['running', 'ready'].include?(current['status']) + end + begin + if terminal && termination.nil? + Process.kill('TERM', pid) + termination = now + elsif termination && now - termination > 5 + Process.kill('KILL', pid) + end + rescue Errno::ESRCH + # The child may exit between the nonblocking wait and signal delivery. + nil + end + sleep(1) + end + ensure + unless reaped + begin + Process.kill('KILL', pid) + rescue Errno::ESRCH + nil + end + Process.waitpid(pid) + end + synchronise do + current = read(id) + if current && current['status'] != 'ready' + if ['running', 'cancelling'].include?(current['status']) + current['status'] = current['status'] == 'cancelling' ? 'cancelled' : 'failed' + current['error'] = 'interrupted' if current['status'] == 'failed' + end + remove_download(current) + remove_partial(id) + current.delete('token') + current['updated_at'] = Time.current.to_f + write(current) + end + end + end + end + + def self.perform(state) + actor = state['actor_id'] && User.find_by(id: state['actor_id']) + actor = nil unless state['admin'] && actor&.admin? && !actor.banned? + HistoryAccess.set(user: actor) do + unless HistoryAccess.authorised_artifact?(state) + synchronise do + current = read(state.fetch('id')) + current['status'] = 'failed' + write(current) + end + return + end + perform_authorised(state) + end + end + + def self.perform_authorised(state) + id = state.fetch('id') + token = nil + begin + checkpoint(id) + exporter = CountsExporter.new(state.fetch('payload'), language_code: state.fetch('language')) + last = Process.clock_gettime(Process::CLOCK_MONOTONIC) + progress = lambda do |processed, total| + now = Process.clock_gettime(Process::CLOCK_MONOTONIC) + if processed.zero? || now - last >= 0.25 || processed == total + checkpoint(id, processed, total) + last = now + end + end + check = lambda do + now = Process.clock_gettime(Process::CLOCK_MONOTONIC) + if now - last >= 0.25 + checkpoint(id) + last = now + end + end + phase = ->(value) { checkpoint(id, phase: value) } + token = CountsExportDownload.prepare(exporter, state.fetch('owner'), + progress:, phase:, check:, identifier: id) + file = CountsExportDownload.find(token, state.fetch('owner')) + raise Cancelled unless file + + synchronise do + current = read(id) + if (!current || current['status'] != 'running' || + current['expires_at'] <= Time.current.to_f) + File.unlink(file.fetch('path')) if File.exist?(file.fetch('path')) + raise Cancelled + end + current.merge!('status' => 'ready', 'token' => token, + 'updated_at' => Time.current.to_f, + 'download_file' => File.basename(file.fetch('path'))) + current['total'] = current['processed'] + write(current) + end + rescue StandardError => error + remove_partial(id) + synchronise do + current = read(id) + if current + unless current['status'] == 'failed' + current['status'] = error.is_a?(Cancelled) ? 'cancelled' : 'failed' + current['error'] = failure_code(error) + end + current['updated_at'] = Time.current.to_f + write(current) + end + end + unless error.is_a?(Cancelled) + Rails.logger.error("counts_export_job #{ id }: #{ error.class }") + end + end + end +end diff --git a/backend/app/services/counts_export_projection.rb b/backend/app/services/counts_export_projection.rb new file mode 100644 index 0000000..6e521de --- /dev/null +++ b/backend/app/services/counts_export_projection.rb @@ -0,0 +1,286 @@ +require 'set' + +# Export identifiers are stable; UI translations are supplied separately as column labels. +class CountsExportProjection + ROOT_KEYS = { 'event_type' => 'event_type', 'resource_id' => 'resource_id', 'version_no' => 'version_no', + 'latest' => 'latest', 'thumbnail' => 'thumbnail_url', + 'thumbnail_base' => 'thumbnail_base', 'title' => 'title', 'url' => 'url', + 'tags' => 'tags', 'parents' => 'parents', 'video_ms' => 'video_ms', + 'original_created_from' => 'original_created_from', + 'original_created_before' => 'original_created_before', + 'user' => 'created_by_user', 'created_at' => 'created_at', + 'name' => 'name', 'names' => 'names', 'category' => 'category', + 'aliases' => 'aliases', 'deprecated' => 'deprecated' }.freeze + CHILD_KEYS = { 'id' => 'id', 'versionNo' => 'version_no', 'kind' => 'kind', + 'name' => 'name', 'category' => 'category', 'sections' => 'sections', + 'names' => 'names', 'aliases' => 'aliases', + 'languageCode' => 'language', 'scriptCode' => 'script', + 'autoGenerated' => 'auto_generated' }.freeze + NAME_FIELDS = ['languageCode', 'scriptCode', 'name', 'aliases', 'autoGenerated'].freeze + CHILDREN = { + 'tags' => ['id', 'versionNo', 'kind', 'name', 'category', 'sections', 'names', 'aliases'], + 'names' => NAME_FIELDS, + 'aliases' => ['languageCode', 'name', 'scriptCode'], + 'parents' => ['id', 'name', 'names'] }.freeze + + def self.default_columns(resource) + fields = { 'posts' => ['title', 'url', 'tags', 'parents', 'video_ms'], + 'tags' => ['name', 'category', 'aliases', 'parents'], + 'nico_tags' => ['name', 'linked_tags'], + 'tag_descriptions' => ['language_code', 'body', 'source_language_code'], + 'tag_groups' => ['name', 'description', 'members'], + 'materials' => ['url', 'tag_id', 'source_kind'] }.fetch(resource) + ['resource_id', 'version_no'] + fields + ['event_type', 'user', 'created_at'] + end + + def self.default_children(resource) + { 'posts' => ['tags.name', 'parents.id'], + 'tags' => ['aliases.name', 'parents.id'], + 'tag_groups' => ['members.tagId'] }.fetch(resource, []) + end + + attr_reader :columns, :headers + + def initialize(params, resource:, summary:, tags:) + params = CountsResourceDefinition.normalise_params(params) + if params.key?('definition') + params = CountsDefaults.for(params.merge('resource' => resource)).merge(params) + end + @params = params + definition = CountsResourceDefinition.new(resource) + name_keys = definition.name_columns.map { _1[:key] } + @root_keys = ROOT_KEYS.merge(name_keys.to_h { [_1, _1] }) + @child_keys = CHILD_KEYS.merge(name_keys.to_h { [_1, _1] }) + @children = CHILDREN.transform_values(&:dup) + ['tags', 'parents'].each { @children[_1] += name_keys } + @complete_names = params.fetch('format', 'json') == 'json' + @resource = resource + @labels = JSON.parse(params.fetch('labels', '{}')) + raise ArgumentError unless @labels.is_a?(Hash) && @labels.length <= 100 + raise ArgumentError unless @labels.all? { |key, value| valid_label?(key, value) } + @summary = summary + allowed = if summary + tags ? ['kind', 'tag_id', 'name', 'deprecated', 'count'] : ['group', 'count'] + else + CountsQuery::COLUMNS.fetch(resource) + (resource == 'tags' ? name_keys : []) + end + defaults = summary ? allowed : self.class.default_columns(resource) + @columns = QueryState.list(params.fetch('columns', defaults.join(' '))) + raise ArgumentError if (@columns - allowed).any? || @columns.uniq != @columns + + @headers = JSON.parse(params.fetch('headers', JSON.generate(@columns))) + unless (@headers.is_a?(Array) && @headers.length == @columns.length && + @headers.all? { _1.is_a?(String) && _1.length <= 200 }) + raise ArgumentError + end + @order = QueryState.list(params.fetch('order', '')) + @sorts = QueryState.list(params.fetch('child_sort', '')).map do |entry| + parts = entry.split(':', -1) + raise ArgumentError unless parts.length == 2 + + path, direction = parts + raise ArgumentError unless ['asc', 'desc'].include?(direction) + + [path, direction] + end + @selected = params.key?('children') ? QueryState.list(params['children']).to_set : + self.class.default_children(resource).to_set + valid = @children.flat_map do |key, fields| + paths(fields, key) + end + raise ArgumentError if @selected && (@selected.to_a - valid).any? + prune = lambda do |fields, prefix| + fields.each do |key| + path = "#{ prefix }.#{ key }" + children = fields_for(key) + next if children.empty? + + prune.call(children, path) + @selected.delete(path) unless children.any? { @selected.include?("#{ path }.#{ _1 }") } + end + end + @children.each { |key, fields| prune.call(fields, key) } + pairs = @columns.zip(@headers).select do |key, _header| + !@children.key?(key) || @children[key].any? { @selected.include?("#{ key }.#{ _1 }") } + end + @columns = pairs.map(&:first) + @headers = pairs.map(&:last) + allowed_sorts = valid.map do |path| + parts = path.split('.') + field = parts.pop + "#{ parts.join('.') }/#{ field }" + end + if ((@sorts.map(&:first) - allowed_sorts).any? || + @sorts.map(&:first).uniq.length != @sorts.length) + raise ArgumentError + end + end + + def metadata + @columns.each_with_index.map do |key, index| + { key: @summary ? key : @root_keys.fetch(key), label: @headers[index] } + end + end + + def project(row) + result = @columns.to_h do |key| + if @summary + value = key == 'group' ? { key: row[:key], name: row[:name] } : row[key.to_sym] + [key, value] + else + value = case key + when 'thumbnail' then row[:thumbnail_base] + else row[@root_keys.fetch(key).to_sym] + end + value = nested(value || [], @children.fetch(key), key) if @children.key?(key) + [@root_keys.fetch(key), value] + end + end + if @complete_names && ((!@summary && @resource == 'tags') || row.key?(:names)) + result['names'] = row[:names] || [] + end + result['aliases'] = row[:aliases] if @complete_names && row.key?(:aliases) + result + end + + def csv_record(record, row) + record.map do |key, value| + key == 'group' ? row[:name] || row[:key] : csv_display(value, key) + end + end + + private + + def valid_label?(key, value) + key.is_a?(String) && value.is_a?(String) && key.length <= 100 && value.length <= 300 + end + + def csv_display(value, key) + if key.start_with?('aliases_') && value.is_a?(Array) + return value.map { fetch(_1, 'name') }.compact.join(' ') + end + return @labels.fetch('bot', '') if key == 'created_by_user' && value.nil? + return @labels.fetch("event_type:#{ value }", value) if key == 'event_type' + return value ? @labels.fetch('latest', 'true') : '' if key == 'latest' + return @labels.fetch("kind:#{ value }", value) if key == 'kind' + return @labels.fetch("category:#{ value }", value) if key == 'category' + if ['language', 'script'].include?(key) && value.is_a?(Hash) + return @labels.fetch("language:#{ value[:code] }", value[:code]) if key == 'language' + return @labels.fetch("#{ key }:#{ value[:code] }", + value[:name] || @labels.fetch('notRegistered', '')) + end + return @labels.fetch("boolean:#{ value }", value) if value == true || value == false + return value.map { csv_display(_1, key) } if value.is_a?(Array) + if value.is_a?(Hash) + return value.to_h do |field, item| + [@labels.fetch(field.to_s, field), csv_display(item, field.to_s)] + end + end + + value + end + + def fields_for(key) + return NAME_FIELDS if key == 'names' + return ['languageCode', 'name', 'scriptCode'] if key == 'aliases' + + [] + end + + def paths(fields, prefix) + fields.flat_map do |key| + path = "#{ prefix }.#{ key }" + [path, *paths(fields_for(key), path)] + end + end + + def fetch(row, key) = row.key?(key) ? row[key] : row[key.to_sym] + + def named_code(row, kind) + code = fetch(row, "#{ kind }_code") + name = fetch(row, "#{ kind }_name") + { code:, name: name.present? && name.to_s.downcase != code.to_s.downcase ? name : nil } + end + + def stable(value) + case value + when Hash + value.stringify_keys.sort.to_h.transform_values { stable(_1) } + when Array + value.map { stable(_1) } + else + value + end + end + + def sort_value(row, key) + return fetch(row, 'language_code') if key == 'languageCode' + return fetch(row, 'script_code') if key == 'scriptCode' + + fetch(row, @child_keys.fetch(key)) + end + + def comparison(left, right) + if left.is_a?(Array) && right.is_a?(Array) && left.length != right.length + return left.length <=> right.length + end + return left <=> right if left.is_a?(Numeric) && right.is_a?(Numeric) + if [true, false].include?(left) && [true, false].include?(right) + return (left ? 1 : 0) <=> (right ? 1 : 0) + end + + a = left.is_a?(Hash) || left.is_a?(Array) ? JSON.generate(stable(left)) : left.to_s + b = right.is_a?(Hash) || right.is_a?(Array) ? JSON.generate(stable(right)) : right.to_s + a <=> b + end + + def sorted(rows, prefix, fields) + sorts = @sorts.filter_map do |path, direction| + parent, key = path.split('/', 2) + next unless parent == prefix + raise ArgumentError unless fields.include?(key) + + [key, direction] + end + rows.each_with_index.sort do |(left, a), (right, b)| + result = 0 + sorts.each do |key, direction| + first = sort_value(left, key) + last = sort_value(right, key) + result = if first.nil? || last.nil? + first.nil? == last.nil? ? 0 : (first.nil? ? 1 : -1) + else + comparison(first, last) * (direction == 'asc' ? 1 : -1) + end + break unless result.zero? + end + result.zero? ? a <=> b : result + end.map(&:first) + end + + def nested(rows, fields, prefix) + keys = fields.select { !@selected || @selected.include?("#{ prefix }.#{ _1 }") } + ordered = (@order + keys.map { "#{ prefix }.#{ _1 }" }).uniq + .filter_map { |path| keys.find { "#{ prefix }.#{ _1 }" == path } } + sorted(rows, prefix, fields).map do |row| + result = ordered.to_h do |key| + stable = @child_keys.fetch(key) + value = case key + when 'languageCode' then named_code(row, 'language') + when 'scriptCode' then named_code(row, 'script') + else fetch(row, stable) + end + if fields_for(key).any? + value = nested(value || [], fields_for(key), "#{ prefix }.#{ key }") + end + [stable, value] + end + if @complete_names && ['tags', 'parents'].include?(prefix) + result['names'] = fetch(row, 'names') || [] + aliases = fetch(row, 'aliases') + result['aliases'] = aliases if aliases + end + result + end + end +end diff --git a/backend/app/services/counts_exporter.rb b/backend/app/services/counts_exporter.rb new file mode 100644 index 0000000..8af545a --- /dev/null +++ b/backend/app/services/counts_exporter.rb @@ -0,0 +1,164 @@ +require 'csv' + +# One read view and a fixed point for the entire download; only one page is held in memory. +class CountsExporter + EXTRA_KEYS = ['basis', 'format', 'resource', 'headers', 'children', 'order', + 'summary_columns', 'nested_json', 'child_sort', 'labels'].freeze + attr_reader :format, :resource + + def self.with_read_snapshot(&block) + # A caller-owned transaction already owns the read view (including fixtures). + # MySQL cannot change isolation inside it. Standalone workers always start RR. + options = ApplicationRecord.connection.transaction_open? ? {} : { isolation: :repeatable_read } + ApplicationRecord.transaction(**options, &block) + end + + def initialize(params, language_code:) + params = CountsResourceDefinition.normalise_params(params) + if params.key?('definition') + @table_export = CountsTableExport.new(params) + @format = @table_export.format + @resource = @table_export.resource + return + end + allowed = (CountsQuery::KEYS + CountsStateQuery::KEYS + EXTRA_KEYS).uniq + QueryState.validate!(params, allowed) + raise ArgumentError unless params.values.all? { _1.is_a?(String) } + + @params = params + @language = language_code + @format = params.fetch('format', 'json') + @resource = params.fetch('resource', 'posts') + @mode = params.fetch('mode', 'summary') + @history = params.fetch('basis', 'state') == 'history' + raise ArgumentError unless ['json', 'csv'].include?(format) + raise ArgumentError unless ['posts', 'tags'].include?(resource) + raise ArgumentError unless ['summary', 'details'].include?(@mode) + raise ArgumentError unless ['state', 'history'].include?(params.fetch('basis', 'state')) + + @query_params = params.except(*EXTRA_KEYS, 'columns', 'page', 'limit') + .merge('mode' => @mode, 'page' => '1', 'limit' => '100') + point = CountsStateQuery.new(@query_params.slice('at').merge('resource' => resource), + language_code: @language).at + @query_params['at'] = point.iso8601(6) + @query_params['axis'] ||= !@history && resource == 'posts' ? 'tags' : 'month' + @projection = CountsExportProjection.new(params, resource:, + summary: @mode == 'summary', + tags: !@history && @query_params['axis'] == 'tags') + if @history + @query_params.delete('kind') + history_query + else + @query_params.delete('latest') + @state_query = CountsStateQuery.new(@query_params.merge('resource' => resource), + language_code: @language).validate_query! + end + end + + def body(progress: nil, phase: nil) + return @table_export.body(progress:, phase:) if @table_export + + Enumerator.new do |output| + self.class.with_read_snapshot do + processed = 0 + first_row = true + started = false + phase&.call('aggregating') + pages do |page| + total = @mode == 'details' ? page[:count] : nil + progress&.call(processed, total) + phase&.call('writing') + unless started + meta = { resource:, mode: @mode, basis: @history ? 'history' : 'state', + at: @query_params['at'], boundary: 'inclusive', + filters: @query_params.except('sort', 'page', 'limit', 'mode'), + sort: export_sort, + child_sort: sort_metadata(@params.fetch('child_sort', '')), + columns: @projection.metadata, + children: @params['children']&.split(' '), + field_order: QueryState.list(@params.fetch('order', '')), timezone: 'UTC', + count: @mode == 'summary' ? page[:group_count] : page[:count], + totals: page[:totals], coverage: page[:coverage] } + output << (format == 'json' ? "{\"metadata\":#{ meta.to_json },\"data\":[" : "\uFEFF") + if format == 'csv' + output << CSV.generate_line( + @projection.headers.map { csv_value(_1) }, row_sep: "\r\n") + end + started = true + end + (page[:versions] || page[:state_rows] || page[:groups] || []).each do |row| + record = @projection.project(row) + if format == 'json' + output << ',' unless first_row + output << record.to_json + else + values = @projection.csv_record(record, row).map { csv_value(_1) } + output << CSV.generate_line(values, row_sep: "\r\n") + end + first_row = false + processed += 1 + progress&.call(processed, total) + end + phase&.call('aggregating') + end + progress&.call(processed, processed) + output << ']}' if format == 'json' + end + end + end + + private + + def export_sort + sort_metadata(@query_params.fetch('sort', default_sort)) + end + + def sort_metadata(value) + value.split(' ').map do |entry| + key, direction = entry.split(':', 2) + { key:, direction: } + end + end + + def default_sort + return 'created_at:desc' if @mode == 'details' + + !@history && @query_params['axis'] == 'tags' ? 'count:desc' : 'group:asc' + end + + def history_query + scope = resource == 'posts' ? PostVersion.with_previous_version : TagVersion.all + CountsQuery.new(scope, resource, @query_params) + end + + def pages + unless @history + @state_query.each_page { yield _1 } + return + end + query = history_query + total = @mode == 'summary' ? nil : query.count + loop do + result = if @mode == 'summary' + query.summary(include_count: @query_params['page'] == '1') + else + { count: total, + versions: CountsVersionRepr.many(query.details, resource:, + language_code: @language, + at: Time.iso8601(@query_params['at']), + history_flg: true) } + end + total ||= result[:group_count] + yield result + break if @query_params['page'].to_i * 100 >= total + + @query_params['page'] = (@query_params['page'].to_i + 1).to_s + query = history_query + end + end + + def csv_value(value) + text = value.is_a?(Hash) || value.is_a?(Array) ? JSON.generate(value) : value.to_s + text.match?(/\A\s*[=+@-]/) ? "'#{ text }" : text + end +end diff --git a/backend/app/services/counts_pivot_layout.rb b/backend/app/services/counts_pivot_layout.rb new file mode 100644 index 0000000..2d9e897 --- /dev/null +++ b/backend/app/services/counts_pivot_layout.rb @@ -0,0 +1,52 @@ +# Coordinates carry their aggregation depth; SQL NULL remains an ordinary axis value. +class CountsPivotLayout + def self.depths(keys, sorts, measures: true) + return [] if keys.length < 2 || sorts.first&.first == 'count' || + sorts.first&.first&.match?(/\Am[0-9]+\z/) + + relevant = sorts.map(&:first).select { keys.include?(_1) || (measures && (_1 == 'count' || + _1.match?(/\Am[0-9]+\z/))) } + order = (relevant + keys).uniq + (1...keys.length).select { order.first(_1).sort == keys.first(_1).sort } + end + + def self.label_key(row_count, column_count, row_depth, column_depth) + total = row_count + column_count + collapsed = total - row_depth - column_depth + return nil if collapsed.zero? + return 'grandTotal' if total >= 3 && collapsed == total + if (collapsed == 1 && + ((row_count > 1 && row_depth < row_count) || + (column_count > 1 && column_depth < column_count))) + return 'subtotal' + end + 'singleTotal' + end + + def self.headers(values, depths, next_values: nil, grand: false, axis_count: nil, + other_count: 0, direction: :row) + result = [] + values.each_with_index do |tuple, index| + result << { values: tuple, kind: 'data' } + # A group continuing on the next page closes only on its final page. + following = values[index + 1] || next_values + depths.reverse_each do |depth| + prefix = tuple.first(depth) + next if following && following.first(depth) == prefix + + result << { values: prefix, kind: 'subtotal' } + end + end + result << { values: [], kind: 'grand_total' } if grand + count = axis_count || values.first&.length || 0 + result.each do |header| + depth = header[:values].length + label = if header[:kind] != 'data' + direction == :row ? label_key(count, other_count, depth, other_count) : + label_key(other_count, count, other_count, depth) + end + header.merge!(depth:, axis_count: count, label_key: label) + end + result + end +end diff --git a/backend/app/services/counts_pivot_totals.rb b/backend/app/services/counts_pivot_totals.rb new file mode 100644 index 0000000..65cc362 --- /dev/null +++ b/backend/app/services/counts_pivot_totals.rb @@ -0,0 +1,91 @@ +# Re-aggregate distinct membership; tag row tuples alone use additive contributions. +class CountsPivotTotals + class TimedOut < ActiveRecord::StatementTimeout; end + def initialize(population, axes, row_headers, column_headers) + @population = population + @axes = axes + @rows = row_headers.group_by { _1[:values].length } + @columns = column_headers.group_by { _1[:values].length } + end + + def call + # Materialise full-cell membership once, including omitted multi-value axes. + membership = @population.scope.except(:select, :order) + joins = @axes.joins + membership = membership.joins(joins) unless joins.empty? + # Retain exactly the records contributing a full cell, including changed_field. + basis = membership.select('v.*').distinct + .limit(18_446_744_073_709_551_615).to_sql + queries = @rows.flat_map do |row_depth, headers| + @columns.filter_map do |column_depth, columns| + next if row_depth == @axes.rows.length && column_depth == @axes.columns.length + + indexes = (0...row_depth).to_a + + (0...column_depth).map { @axes.rows.length + _1 } + expressions = indexes.map { @axes.expressions[_1] } + scope = @population.scope.except(:select, :order, :where, :from).from('population_members v') + additive = @axes.rows.any? { _1['field'] == 'tags' } + retained = additive ? (0...@axes.rows.length).to_a | indexes : indexes + joins = @axes.joins(retained.map { @axes.all[_1].fetch('field') }) + scope = scope.joins(joins) unless joins.empty? + predicates = [] + if row_depth.positive? + tuples = headers.map { _1[:values] }.uniq + predicate = tuples.map { Array.new(row_depth) { |i| + "a#{ indexes[i] } <=> ?" + }.join(' AND ') }.map { "(#{ _1 })" }.join(' OR ') + predicates << @population.definition.model.sanitize_sql_array([ + tuples.empty? ? 'FALSE' : "(#{ predicate })", *tuples.flatten]) + end + if column_depth.positive? + tuples = columns.map { _1[:values] }.uniq + predicate = tuples.map { Array.new(column_depth) { |i| + "a#{ indexes[row_depth + i] } <=> ?" + }.join(' AND ') }.map { "(#{ _1 })" }.join(' OR ') + predicates << @population.definition.model.sanitize_sql_array([ + tuples.empty? ? 'FALSE' : "(#{ predicate })", *tuples.flatten]) + end + # Filter projected coordinates, not repeated masking/time expressions. + # Each member still participates in the complete DISTINCT population. + selection = predicates.empty? ? '' : " WHERE #{ predicates.join(' AND ') }" + grouped = indexes.each_with_index.map { |key, index| + "a#{ key } AS v#{ index }" + } + # Fixed-width result coordinates, independent of NULL and of translated labels. + padding = (expressions.length...@axes.all.length).map { "NULL AS v#{ _1 }" } + fields = ["#{ row_depth } AS row_depth", "#{ column_depth } AS column_depth"] + + grouped + padding + ['count'] + if additive + # Sum full row tuples, not page cells; preserve DISTINCT inside each tuple. + inner = @axes.grouped_sql(scope, retained) + coordinates = indexes.map { "a#{ _1 }" } + projected = coordinates.each_with_index.map { |key, index| "#{ key } AS v#{ index }" } + output = ["#{ row_depth } AS row_depth", "#{ column_depth } AS column_depth"] + + projected + padding + ['SUM(count) AS count'] + grouping = coordinates.empty? ? '' : " GROUP BY #{ coordinates.join(', ') }" + "SELECT #{ output.join(', ') } FROM (#{ inner }) leaf#{ selection }#{ grouping }" + else + inner = @axes.grouped_sql(scope, indexes) + "SELECT #{ fields.join(', ') } FROM (#{ inner }) grouped#{ selection }" + end + end + end + return [] if queries.empty? + + sql = queries.map { "(#{ _1 })" }.join(' UNION ALL ') + statement = "WITH population_members AS (#{ basis }) " + + "SELECT /*+ #{ CountsTableQuery::SQL_HINTS.join(' ') } */ * FROM (#{ sql }) totals" + @population.definition.model.connection.select_all(statement).map do |record| + row_depth = record['row_depth'].to_i + column_depth = record['column_depth'].to_i + { row: (0...row_depth).map { record["v#{ _1 }"] }, + column: (0...column_depth).map { record["v#{ row_depth + _1 }"] }, + row_depth:, column_depth:, + label_key: CountsPivotLayout.label_key(@axes.rows.length, @axes.columns.length, + row_depth, column_depth), + count: record['count'].to_i } + end + rescue ActiveRecord::StatementTimeout + raise TimedOut, 'Counts pivot totals exceeded the query time limit' + end +end diff --git a/backend/app/services/counts_population.rb b/backend/app/services/counts_population.rb new file mode 100644 index 0000000..3776444 --- /dev/null +++ b/backend/app/services/counts_population.rb @@ -0,0 +1,240 @@ +# Population first, then filters. Previous versions are joined before period filtering. +class CountsPopulation + attr_reader :definition, :params, :at, :history, :scope, :timezone + + def initialize(params, authorisation_flg: true) + @authorisation_flg = authorisation_flg + @params = params + @definition = CountsResourceDefinition.new(params.fetch('resource', 'posts')) + @history = params.fetch('definition', 'latest') == 'history' + raise ArgumentError unless ['latest', 'history'].include?(params.fetch('definition', 'latest')) + + @timezone = TZInfo::Timezone.get(params.fetch('timezone', 'UTC')) + @at = timestamp(params['execution_at']) || Time.current.utc + raise ArgumentError if at > Time.current + 1.second + + @scope = definition.model.all + make_population + @scope = HistoryAccess.visible(@scope, 'v') if @authorisation_flg + apply_filters + rescue TZInfo::InvalidTimezoneIdentifier + raise ArgumentError + end + + def timestamp(value, end_date: false) + return nil if value.nil? || value == '' + raise ArgumentError unless value.is_a?(String) + + if value.match?(/\A\d{4}-\d{2}-\d{2}\z/) + date = Date.iso8601(value) + date += 1 if end_date + timezone.local_to_utc(Time.utc(date.year, date.month, date.day)) + else + raise ArgumentError unless value.match?(/(?:Z|[+-]\d{2}:\d{2})\z/) + + time = Time.iso8601(value).utc + raise ArgumentError unless time.year.between?(1900, 2100) + + time + end + rescue TZInfo::AmbiguousTime, TZInfo::PeriodNotFound, Date::Error + raise ArgumentError + end + + def latest_at_sql(row, cutoff) + keys = definition.keys.map { "newer.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ') + bound = definition.model.connection.quote(cutoff) + <<~SQL.squish + NOT EXISTS (SELECT 1 FROM #{ definition.table } newer + WHERE #{ keys } AND newer.version_no > #{ row }.version_no + AND newer.#{ definition.time_column } <= #{ bound }) + SQL + end + + def current_latest_sql(row = 'v') + keys = definition.keys.map { "newer.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ') + <<~SQL.squish + NOT EXISTS (SELECT 1 FROM #{ definition.table } newer + WHERE #{ keys } AND newer.version_no > #{ row }.version_no) + SQL + end + + def changed_sql(key) + column = definition.column(key) + "(previous.id IS NOT NULL AND NOT (v.#{ column } <=> previous.#{ column }))" + end + + # Aggregate sources are unrestricted; CountsAxes masks their output coordinates. + # Never serialise raw records from an unrestricted population. + def access_sql(model, row) + @authorisation_flg ? HistoryAccess.sql(model, row) : 'TRUE' + end + + def axis_condition(scope, entry) + raise CountsAxes::PrivateCondition if entry['value'] == CountsAxes::MASKED_VALUE + + key = entry.fetch('field') + raise ArgumentError unless definition.axes(history).include?(key) + value = entry['value'] + unless value.nil? + valid = case definition.fields[key] + when :integer + value.is_a?(Integer) || (value.is_a?(String) && value.match?(/\A-?[0-9]+\z/)) + when :boolean then [true, false].include?(value) + else value.is_a?(String) + end + raise ArgumentError unless valid + end + if entry.key?('grain') + unless definition.fields[key] == :datetime && CountsAxes::GRAINS.include?(entry['grain']) + raise ArgumentError + end + end + if key == 'tags' + scope = scope.where(snapshot_membership(entry.fetch('value'))) + elsif key == 'changed_field' + raise ArgumentError unless history && definition.changes.include?(entry['value']) + + scope = scope.where(Arel.sql(changed_sql(entry['value']))) + elsif definition.fields[key] == :datetime && entry['grain'] + scope = scope.where(definition.expression(key) => nil) if entry['value'].nil? + if entry['value'] + expression = CountsAxes.time_expression(definition.expression(key), + entry['grain'], timezone) + scope = scope.where("#{ expression } = ?", entry['value']) + end + else + raise ArgumentError unless definition.axes(history).include?(key) + + scope = scope.where(definition.expression(key) => entry['value']) + end + scope + end + + private + + def make_population + table = definition.table + @scope = @scope.from("#{ table } v").select('v.*') + previous = ['rows', 'cols'].any? do |key| + QueryState.axes(params.fetch(key, '')).any? { _1['field'] == 'changed_field' } + end || (QueryState.records(params, 'drill') + QueryState.records(params, 'conditions')) + .any? { _1['field'] == 'changed_field' } + if previous + keys = definition.keys.map { "previous.#{ _1 } = v.#{ _1 }" }.join(' AND ') + @scope = @scope.joins("LEFT JOIN #{ table } previous ON #{ keys } " + + 'AND previous.version_no = v.version_no - 1 AND ' + + access_sql(definition.model, 'previous')) + end + @scope = @scope.where("v.#{ definition.time_column } <= ?", at) + unless history + point = timestamp(params['at']) || at + raise ArgumentError if point > at + + @scope = @scope.where("v.#{ definition.time_column } <= ?", point) + .where(Arel.sql(latest_at_sql('v', point))).where.not('v.event_type' => 'discard') + return + end + first = timestamp(params['from']) + last = timestamp(params['through'], end_date: true) + raise ArgumentError if first && last && first > last + + @scope = @scope.where('v.created_at >= ?', first) if first + if last + operator = params['through'].length == 10 ? '<' : '<=' + @scope = @scope.where("v.created_at #{ operator } ?", last) + end + end + + def apply_filters + ['resource_id', 'user'].each do |key| + value = params[key == 'user' ? 'user_id' : key] + next if value.nil? || value == '' + raise ArgumentError unless value.match?(/\A[0-9]+\z/) + + @scope = @scope.where("#{ definition.expression(key) } = ?", value.to_i) + end + apply_tags if params['tags'].present? || params['tag'].present? + conditions = QueryState.records(params, 'conditions') + drills = QueryState.records(params, 'drill') + raise ArgumentError unless drills.length <= 100 + drills.each do |entry| + raise ArgumentError unless (entry.keys - ['field', 'grain', 'value']).empty? + + condition = entry.merge('operator' => 'axis_eq') + conditions << condition unless conditions.include?(condition) + end + raise ArgumentError unless conditions.is_a?(Array) && conditions.length <= 100 + + values = QueryState.records(params, 'condition_values') + values.each do |entry| + index = entry.fetch('condition') + raise ArgumentError unless index.between?(0, conditions.length - 1) + raise ArgumentError unless conditions[index]['values'] == true + end + evaluated = [] + conditions.each_with_index do |entry, index| + if entry['operator'] == 'axis_eq' + next if evaluated.include?(entry) + + evaluated << entry + end + if entry.key?('values') + raise ArgumentError unless entry['values'] == true + + entry = entry.merge('values' => values.select { _1['condition'] == index } + .map { _1.fetch('value') }) + end + apply_condition(entry) + end + + end + + def apply_condition(entry) + @condition ||= CountsCondition.new(self) + @scope = @condition.apply(@scope, entry) + end + + def snapshot_membership(value) + kind, identifier = value.to_s.split(':', 2) + raise ArgumentError unless ['internal', 'external'].include?(kind) + raise ArgumentError unless identifier&.match?(/\A[1-9]\d*\z/) + + key = kind == 'internal' ? 'tag_id' : 'external_tag_id' + definition.model.sanitize_sql_array([ + "JSON_CONTAINS(v.tags_json, JSON_OBJECT('#{ key }', CAST(? AS UNSIGNED)))", + identifier.to_i]) + end + + def apply_tags + raise ArgumentError unless definition.resource == 'posts' + + if params['tag'].present? + @scope = @scope.where(snapshot_membership( + "#{ params.fetch('kind', 'internal') }:#{ params['tag'] }")) + end + return unless params['tags'].present? + + raw = params['tags'] + tokens = raw.split.reject { _1.downcase == 'type:or' } + locale = Locale.find_by(code: params.fetch('locale', 'ja')) + raise ArgumentError unless locale + + literals = PostTagSearch.literals(locale, tokens.join(' ')) + identities = PostTagSearch.references_for(locale, literals) + predicates = literals.map do |literal| + # Resolve current names/aliases exactly as normal post search does. Only + # membership comes from history; snapshot names never identify a tag. + internal, external = identities.fetch(literal[:name]) + parts = { 'internal' => internal, 'external' => external }.flat_map do |kind, ids| + Array(ids).map { |id| snapshot_membership("#{ kind }:#{ id }") } + end + expression = parts.empty? ? 'FALSE' : "(#{ parts.join(' OR ') })" + literal[:negative] ? "NOT #{ expression }" : expression + end + return if predicates.empty? + + any = params['match'] == 'any' || raw.split.any? { _1.downcase == 'type:or' } + @scope = @scope.where(Arel.sql("(#{ predicates.join(any ? ' OR ' : ' AND ') })")) + end +end diff --git a/backend/app/services/counts_query.rb b/backend/app/services/counts_query.rb new file mode 100644 index 0000000..3525d2e --- /dev/null +++ b/backend/app/services/counts_query.rb @@ -0,0 +1,201 @@ +# Read-only history queries. SQL identifiers and expressions are server-owned. +class CountsQuery + AXES = ['month', 'user'].freeze + COMMON_COLUMNS = ['resource_id', 'version_no', 'latest', 'event_type', 'user', 'created_at'].freeze + COLUMNS = { + 'posts' => COMMON_COLUMNS + [ + 'thumbnail', 'thumbnail_base', 'title', 'url', 'tags', 'parents', 'video_ms', + 'original_created_from', 'original_created_before'], + 'tags' => COMMON_COLUMNS + [ + 'name', 'names', 'category', 'aliases', 'parents', 'deprecated'] }.freeze + KEYS = [ + 'locale', 'mode', 'axis', 'page', 'limit', 'from', 'before', 'latest', + 'user_id', 'resource_id', 'tag', 'columns', 'sort', 'at'].freeze + + attr_reader :page, :limit, :mode, :axis + + def initialize(scope, resource, params, latest_expression: nil) + params = CountsResourceDefinition.normalise_params(params) + @table = scope.klass.table_name + @resource = resource + @id_column = resource == 'posts' ? 'post_id' : 'tag_id' + @params = params + @latest_expression = latest_expression + scope = HistoryAccess.visible(scope) + @scope = scope.except(:select, :order).joins( + "LEFT JOIN users counts_actor ON counts_actor.id = #{ @table }.created_by_user_id") + raise ArgumentError unless (params.keys - KEYS).empty? + raise ArgumentError unless params.values.all? { _1.is_a?(String) } + + @mode = params.fetch('mode', 'summary') + @axis = params.fetch('axis', 'month') + raise ArgumentError unless ['summary', 'details'].include?(mode) && AXES.include?(axis) + + @page = integer('page', 1) + @limit = integer('limit', 20) + raise ArgumentError if limit > 100 || page > 1_000_000 + + columns = QueryState.list(params.fetch('columns', COLUMNS.fetch(resource).join(' '))) + if columns.empty? || columns.uniq != columns || (columns - COLUMNS.fetch(resource)).any? + raise ArgumentError + end + @sort = parse_sort + apply_filters + end + + def relation = @scope + + def count = @scope.count + + def latest_expression + return @latest_expression if @latest_expression + + at = timestamp('at') + cutoff = at ? "AND newer.created_at <= #{ @scope.klass.connection.quote(at) }" : '' + <<~SQL.squish + NOT EXISTS (SELECT 1 FROM #{ @table } newer + WHERE newer.#{ @id_column } = #{ @table }.#{ @id_column } + AND newer.version_no > #{ @table }.version_no #{ cutoff }) + SQL + end + + def details + @scope.select("#{ @table }.*", Arel.sql("(#{ latest_expression }) AS counts_latest")) + .order(Arel.sql(order_sql(detail_expressions, ["#{ @table }.id DESC"]))) + .limit(limit).offset((page - 1) * limit) + end + + def summary(include_count: true) + expression = axis == 'month' ? "DATE_FORMAT(#{ @table }.created_at, '%Y-%m')" : + "#{ @table }.created_by_user_id" + group_order = axis == 'user' ? 'MAX(counts_actor.name)' : expression + sql = @scope.group(Arel.sql(expression)).select(Arel.sql( + "#{ expression } AS group_key, COUNT(*) AS record_count, " + + "#{ group_order } AS group_name")).to_sql + connection = @scope.klass.connection + source = "FROM (#{ sql }) counts_groups" + groups = include_count ? connection.select_value("SELECT COUNT(*) #{ source }") : nil + expressions = { 'group' => ['group_name'], 'count' => ['record_count'] } + order = order_sql(expressions, ['group_key IS NULL ASC', 'group_key ASC']) + rows = connection.select_all( + "SELECT group_key, group_name, record_count #{ source } " + + "ORDER BY #{ order } LIMIT #{ limit } OFFSET #{ (page - 1) * limit }") + { group_count: groups&.to_i, + groups: rows.map do |row| + { key: row['group_key'], + name: axis == 'user' ? row['group_name'] : nil, + count: row['record_count'].to_i } + end } + end + + private + + def parse_sort + value = @params.fetch('sort', mode == 'details' ? 'created_at:desc' : 'group:asc') + return [] if value.empty? + + values = QueryState.list(value).map { _1.split(':', -1) } + allowed = mode == 'details' ? COLUMNS.fetch(@resource) - ['thumbnail'] : ['group', 'count'] + invalid = values.any? do |key, direction, *extra| + extra.any? || !allowed.include?(key) || !['asc', 'desc'].include?(direction) + end + if (values.empty? || values.length > allowed.length || invalid || + values.map(&:first).uniq.length != values.length) + raise ArgumentError + end + values + end + + # NULL always sorts last, including descending keys. The final key is unique. + def order_sql(expressions, final) + (@sort.flat_map do |key, direction| + expressions.fetch(key).flat_map do |expression| + ["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"] + end + end + final).join(', ') + end + + def detail_expressions + common = { 'resource_id' => ["#{ @table }.#{ @id_column }"], + 'version_no' => ["#{ @table }.version_no"], + 'latest' => ["(#{ latest_expression })"], + 'user' => ['counts_actor.name', "#{ @table }.created_by_user_id"], + 'created_at' => ["#{ @table }.created_at"] } + if @resource == 'posts' + fields = [ + 'title', 'url', 'thumbnail_base', 'video_ms', + 'original_created_from', 'original_created_before'] + common.merge(fields.to_h { [_1, ["#{ @table }.#{ _1 }"]] }).merge( + 'tags' => json_order('post_versions.tags_json'), + 'parents' => list_order('post_versions.parent_post_ids')) + else + common.merge( + 'name' => ['tag_versions.name'], + 'names' => json_order('tag_versions.names'), + 'category' => ['tag_versions.category'], + 'aliases' => json_order( + "JSON_EXTRACT(tag_versions.names, '$.*.aliases[*]')"), + 'parents' => list_order('tag_versions.parent_tag_ids'), + 'deprecated' => ['tag_versions.deprecated_at']) + end + end + + # Multi-valued parent cells sort by cardinality, then binary snapshot text. + def json_order(column) + [ + "COALESCE(JSON_LENGTH(#{ column }), 0)", + "COALESCE(CAST(#{ column } AS CHAR), '[]') COLLATE utf8mb4_bin"] + end + + def list_order(column) + [ + "CASE WHEN #{ column } = '' THEN 0 ELSE " + + "1 + LENGTH(#{ column }) - LENGTH(REPLACE(#{ column }, ' ', '')) END", + "#{ column } COLLATE utf8mb4_bin"] + end + + def integer(key, default = nil) + value = @params[key].presence + return default if value.nil? && !default.nil? + raise ArgumentError unless value&.match?(/\A[1-9][0-9]{0,9}\z/) + + value.to_i + end + + def timestamp(key) + value = @params[key].presence + return nil unless value + pattern = /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d{1,6})?)?Z\z/ + raise ArgumentError unless value.match?(pattern) + + Date.iso8601(value.first(10)) + Time.iso8601(value.length == 17 ? value.sub(/Z\z/, ':00Z') : value) + end + + def apply_filters + at = timestamp('at') + first = timestamp('from') + last = timestamp('before') + raise ArgumentError if first && last && first >= last + raise ArgumentError unless ['0', '1'].include?(@params.fetch('latest', '1')) + + @scope = @scope.where("#{ @table }.created_at <= ?", at) if at + @scope = @scope.where(Arel.sql(latest_expression)) unless @params['latest'] == '0' + @scope = @scope.where("#{ @table }.created_at >= ?", first) if first + @scope = @scope.where("#{ @table }.created_at < ?", last) if last + if @params['resource_id'].present? + @scope = @scope.where(@table => { @id_column => integer('resource_id') }) + end + if @params['user_id'].present? + user = @params['user_id'] == 'none' ? nil : integer('user_id') + @scope = @scope.where(@table => { created_by_user_id: user }) + end + return unless @params['tag'].present? + raise ArgumentError unless @resource == 'posts' + + id = integer('tag') + @scope = @scope.where( + "JSON_CONTAINS(post_versions.tags_json, JSON_OBJECT('tag_id', CAST(? AS UNSIGNED))) OR " + + "JSON_CONTAINS(prev.tags_json, JSON_OBJECT('tag_id', CAST(? AS UNSIGNED)))", id, id) + end +end diff --git a/backend/app/services/counts_resource_definition.rb b/backend/app/services/counts_resource_definition.rb new file mode 100644 index 0000000..448c3b2 --- /dev/null +++ b/backend/app/services/counts_resource_definition.rb @@ -0,0 +1,309 @@ +# Server-owned version identities, attributes and field types. No client SQL is accepted. +class CountsResourceDefinition + # Compatibility belongs at the request boundary, not in field definitions. + def self.normalise_params(params) + aliases = nil + predicates = nil + scope = params['name_scope'] + raise ArgumentError if scope && !['locale', 'language'].include?(scope) + params.each_with_object({}) do |(key, value), result| + result[key] = value + next unless value.is_a?(String) + + if ['columns', 'order', 'sort', 'rows', 'cols', 'axis'].include?(key) + result[key] = value.split(/ /, -1).map { |token| + token.sub(/\Ais_latest(?=:|\z)/, 'latest') + }.join(' ') + elsif (key.match?(/\A(?:conditions|drill)\.[0-9]+\.field\z/) && + value == 'is_latest') + result[key] = 'latest' + end + if (['columns', 'children', 'order', 'sort', 'child_sort'].include?(key) && + value.match?(/(?:name|aliases)_[a-z]+_[a-z0-9_]+/)) + aliases ||= new('tags').column_aliases.to_h { [_1[:key], _1[:value]] } + result[key] = result[key].gsub(/(?:name|aliases)_[a-z0-9_]+/) { aliases.fetch(_1, _1) } + end + if (scope != 'language' && key.match?(/\Aconditions\.[0-9]+\.field\z/) && + value.match?(/(?:name|aliases)_[a-z0-9_]+/)) + predicates ||= new('tags').condition_aliases.to_h { [_1[:key], _1[:value]] } + result[key] = value.gsub(/(?:name|aliases)_[a-z0-9_]+/) { predicates.fetch(_1, _1) } + end + end + end + + COMMON = { 'resource_id' => :integer, 'version_no' => :integer, + 'event_type' => :enum, 'user' => :integer, 'created_at' => :datetime }.freeze + DEFINITIONS = { + 'posts' => [PostVersion, ['post_id'], { + 'title' => :string, 'url' => :string, 'thumbnail_base' => :string, + 'tags_json' => :json, 'parent_post_ids' => :string, 'video_ms' => :integer, + 'original_created_from' => :datetime, 'original_created_before' => :datetime }], + 'tags' => [TagVersion, ['tag_id'], { + 'name' => :string, 'names' => :json, 'category' => :enum, 'aliases' => :string, + 'parent_tag_ids' => :string, 'deprecated_at' => :datetime }], + 'nico_tags' => [NicoTagVersion, ['tag_id'], { + 'name' => :string, 'linked_tags' => :string }], + 'tag_descriptions' => [TagDescriptionVersion, ['tag_id', 'language_code'], { + 'language_code' => :enum, 'body' => :string, + 'source_language_code' => :enum, 'source_version_no' => :integer }], + 'tag_groups' => [TagGroupVersion, ['tag_group_id'], { + 'name' => :string, 'description' => :string, + 'members' => :json, 'placements' => :json }], + 'materials' => [MaterialVersion, ['material_id'], { + 'url' => :string, 'parent_id' => :integer, 'tag_id' => :integer, + 'tag_name' => :string, 'tag_category' => :enum, + 'discarded_at' => :datetime, 'updated_at' => :datetime, + 'updated_by_user_id' => :integer, 'export_paths_json' => :json, + 'file_blob_id' => :integer, 'file_filename' => :string, + 'file_content_type' => :string, 'file_byte_size' => :integer, + 'file_checksum' => :string, 'file_sha256' => :string, + 'source_kind' => :enum, 'source_uri' => :string, 'source_path' => :string, + 'source_file_id' => :string, 'normalized_source_key' => :string }] }.freeze + OPERATORS = { + string: ['eq', 'ne', 'like', 'in', 'not_in'], + enum: ['eq', 'ne', 'in', 'not_in'], boolean: ['eq', 'ne', 'in', 'not_in'], + integer: ['eq', 'ne', 'gt', 'lt', 'ge', 'le', 'range', 'in', 'not_in'], + datetime: ['eq', 'ne', 'gt', 'lt', 'ge', 'le', 'range', 'in', 'not_in'], + interval: ['range'] }.freeze + LABELS = { + 'resource_id' => 'counts:columns.resourceId', 'version_no' => 'counts:columns.version', + 'latest' => 'counts:columns.latest', 'event_type' => 'counts:eventType', + 'user' => 'history:actor', 'created_at' => 'counts:columns.changedAt', + 'thumbnail' => 'counts:columns.thumbnailDisplay', + 'thumbnail_base' => 'counts:columns.thumbnailURL', + 'title' => 'counts:columns.postTitle', 'url' => 'counts:columns.postURL', + 'tags' => 'counts:columns.tags', 'tags_json' => 'counts:columns.tags', + 'video_ms' => 'shared:labels.duration', 'name' => 'counts:columns.name', + 'names' => 'counts:columns.names', 'category' => 'common:categoryLabel', + 'aliases' => 'tags:aliases', 'deprecated' => 'counts:columns.deprecated', + 'original_created_from' => 'counts:columns.originalFrom', + 'original_created_before' => 'counts:columns.originalBefore', + 'language_code' => 'counts:columns.language' }.freeze + CHILD_LABELS = { + 'id' => 'counts:columns.resourceId', 'versionNo' => 'history:version', + 'kind' => 'counts:columns.kind', 'name' => 'history:name', + 'category' => 'common:categoryLabel', 'sections' => 'counts:columns.sections', + 'names' => 'counts:columns.names', 'aliases' => 'tags:aliases', + 'languageCode' => 'counts:columns.language', + 'scriptCode' => 'history:nameChanges.script', + 'autoGenerated' => 'history:nameChanges.automatic', + 'tagId' => 'counts:attributes.tag_id', 'tagGroupId' => 'counts:resources.tag_groups', + 'sectionName' => 'counts:attributes.section_name', + 'description' => 'counts:attributes.description', + 'children' => 'counts:attributes.children', 'profile' => 'counts:attributes.profile', + 'path' => 'counts:attributes.path' }.freeze + + attr_reader :model, :keys, :fields, :resource + + def initialize(resource) + @resource = resource + @model, @keys, attributes = DEFINITIONS.fetch(resource) { raise ArgumentError } + @fields = COMMON.merge(attributes) + end + + def table = model.table_name + def time_column = 'created_at' + def column(key) + raise ArgumentError unless fields.key?(key) + + { 'resource_id' => keys.first, 'user' => 'created_by_user_id' }.fetch(key, key) + end + + def expression(key, row = 'v') = "#{ row }.#{ column(key) }" + def changes = fields.keys - COMMON.keys - ['updated_at', 'updated_by_user_id'] + def axes(history) + fields.reject { |_key, type| type == :json }.keys + + (resource == 'posts' ? ['tags'] : []) + (history ? ['changed_field'] : []) + end + + def metadata + fields.map do |key, type| + { key:, type:, nullable: model.columns_hash.fetch(column(key)).null, + operators: OPERATORS.fetch(type, []) } + end + end + + def field_label(key) = LABELS.fetch(key, "counts:attributes.#{ key }") + + def self.name_key(code) = "name_#{ code.to_s.downcase.tr('-', '_') }" + def self.aliases_key(code) = "aliases_#{ code.to_s.downcase.tr('-', '_') }" + + def name_columns + @name_columns ||= begin + codes = Language.select(:code).order(:created_at, :code).map do |language| + { code: language.code, language_code: language.code } + end + names = codes.map do |locale| + { key: self.class.name_key(locale[:code]), label: 'counts:localeName', + name_locale: locale } + end + aliases = codes.map do |locale| + { key: self.class.aliases_key(locale[:code]), label: 'counts:localeAliases', + name_locale: locale, multiple_values: true } + end + names + aliases + end + end + + # Old locale-qualified predicates keep their script restriction at the input boundary. + def legacy_name_columns + @legacy_name_columns ||= Locale.select(:code, :language_code, :script_code) + .order(:code).flat_map do |locale| + codes = { code: locale.code, language_code: locale.language_code, + script_code: locale.script_code } + code = locale.code == locale.language_code ? "#{ locale.code }_#{ locale.script_code }" : + locale.code + [{ key: self.class.name_key(code), label: 'counts:localeName', + name_locale: codes, legacy: true }, + { key: self.class.aliases_key(code), label: 'counts:localeAliases', + name_locale: codes, multiple_values: true, legacy: true }] + end + end + + def column_aliases + legacy_name_columns.map do |field| + language = field[:name_locale][:language_code] + value = if field[:multiple_values] + self.class.aliases_key(language) + else self.class.name_key(language) + end + { key: field[:key], value: } + end + end + + def condition_aliases + legacy_name_columns.filter_map do |field| + code = field[:name_locale][:code] + key = field[:multiple_values] ? self.class.aliases_key(code) : self.class.name_key(code) + next if key == field[:key] + + { key:, value: field[:key] } + end + end + + def name_column(key) + name_columns.find { _1[:key] == key } || legacy_name_columns.find { _1[:key] == key } + end + + def name_expression(key, row = 'v') + if key.start_with?('aliases_') + source, value = alias_source(key, row) + return "CAST(COALESCE((SELECT JSON_ARRAYAGG(#{ value }) OVER (" + + 'ORDER BY locale_alias.ordinal ROWS BETWEEN UNBOUNDED PRECEDING ' + + "AND UNBOUNDED FOLLOWING) FROM #{ source } WHERE #{ value } IS NOT NULL " + + 'LIMIT 1), JSON_ARRAY()) AS CHAR) COLLATE utf8mb4_bin' + end + locale = name_column(key).fetch(:name_locale) + connection = model.connection + path = connection.quote("$.#{ locale[:language_code].to_json }") + value = "JSON_UNQUOTE(NULLIF(JSON_EXTRACT(#{ row }.names, " + + "CONCAT(#{ path }, '.name')), CAST('null' AS JSON)))" + return value unless locale[:script_code] + + script = connection.quote(locale[:script_code]) + "CASE WHEN LOWER(JSON_UNQUOTE(JSON_EXTRACT(#{ row }.names, " + + "CONCAT(#{ path }, '.script_code')))) = LOWER(#{ script }) " + + "THEN #{ value } END" + end + + def alias_source(key, row = 'v') + locale = name_column(key).fetch(:name_locale) + connection = model.connection + path = connection.quote("$.#{ locale[:language_code].to_json }.aliases[*]") + source = "JSON_TABLE(COALESCE(#{ row }.names, JSON_OBJECT()), #{ path } " + + "COLUMNS (ordinal FOR ORDINALITY, name TEXT PATH '$.name', " + + "script TEXT PATH '$.script_code')) locale_alias" + return [source, 'locale_alias.name'] unless locale[:script_code] + + script = connection.quote(locale[:script_code]) + value = "CASE WHEN LOWER(locale_alias.script) = LOWER(#{ script }) " + + 'THEN locale_alias.name END' + [source, value] + end + + def output_fields + content = case resource + when 'posts' + ['thumbnail', 'title', 'url', 'thumbnail_base', 'tags', 'parents', 'video_ms', + 'original_created_from', 'original_created_before'] + when 'tags' then ['name', 'names', 'category', 'aliases', 'parents', 'deprecated'] + else fields.keys - COMMON.keys + end + keys = ['resource_id', 'version_no', 'latest'] + (content - ['thumbnail']) + + ['event_type', 'user', 'created_at'] + keys.unshift('thumbnail') if content.include?('thumbnail') + keys.flat_map do |key| + next name_columns if key == 'name' && resource == 'tags' + next [] if key == 'names' && resource == 'tags' + next [] if key == 'aliases' && resource == 'tags' + label = if key == 'parents' + resource == 'tags' ? 'tags:parents' : 'shared:labels.parentPosts' + else field_label(key) + end + field = { key:, label: } + children = child_fields(key) + field[:children] = children if children.any? + [field] + end + end + + def child_fields(key, depth = 4) + keys = case key + when 'tags' then ['id', 'versionNo', 'kind', 'name', 'category', 'sections', 'aliases'] + when 'names' then ['languageCode', 'scriptCode', 'name', 'aliases', 'autoGenerated'] + when 'aliases' then ['languageCode', 'name', 'scriptCode'] + when 'parents' then resource == 'tags' ? ['id', 'name'] : ['id'] + when 'members' then ['tagId', 'tagGroupId', 'versionNo'] + when 'export_paths_json' then ['profile', 'path'] + when 'placements', 'children' + scalar = ['sectionName', 'description', 'tagId', 'tagGroupId'] + scalar + (depth > 0 ? ['children'] : []) + else [] + end + keys.flat_map do |child| + next [] if child == 'aliases' && key == 'tags' + if child == 'name' && ['tags', 'parents'].include?(key) + next name_columns + end + label = key == 'members' && child == 'tagId' ? + 'counts:columns.resourceId' : CHILD_LABELS.fetch(child) + field = { key: child, label: } + children = child_fields(child, child == 'children' ? depth - 1 : depth) + # Language is attached to top-level aliases, not to aliases inside a language entry. + if child == 'aliases' && key == 'names' + children = children.reject { _1[:key] == 'languageCode' } + end + field[:children] = children if children.any? + [field] + end + end + + def condition_fields + fields = output_fields + if resource == 'tags' + fields = fields.map do |field| + next field unless field[:key] == 'parents' + + field.merge(children: field[:children] + legacy_name_columns + [ + { key: 'name', label: 'history:name' }, + { key: 'names', label: 'counts:columns.names', children: child_fields('names') }]) + end + fields += legacy_name_columns + [{ key: 'name', label: 'counts:columns.name' }, + { key: 'names', label: 'counts:columns.names', children: child_fields('names') }, + { key: 'aliases', label: 'tags:aliases', children: child_fields('aliases') }] + end + if resource == 'posts' + fields = fields.map do |field| + next field unless field[:key] == 'tags' + + field.merge(children: field[:children] + legacy_name_columns + [ + { key: 'name', label: 'history:name' }, + { key: 'names', label: 'counts:columns.names', children: child_fields('names') }, + { key: 'aliases', label: 'tags:aliases', children: child_fields('aliases') }]) + end + end + fields + end + + def filter_metadata = @filter_metadata ||= CountsCondition.fields(self) +end diff --git a/backend/app/services/counts_state_query.rb b/backend/app/services/counts_state_query.rb new file mode 100644 index 0000000..b141718 --- /dev/null +++ b/backend/app/services/counts_state_query.rb @@ -0,0 +1,370 @@ +# Restores one recorded version per resource before applying record filters. +class CountsStateQuery + KEYS = ['locale', 'resource', 'mode', 'at', 'from', 'before', 'grain', + 'kind', 'tag', 'page', 'limit', 'sort', 'axis', 'user_id', 'resource_id'].freeze + MAX_PERIODS = 366 + + attr_reader :at + + def initialize(params, language_code:) + raise ArgumentError unless (params.keys - KEYS).empty? + raise ArgumentError unless params.values.all? { _1.is_a?(String) } + + @params = params + @connection = ApplicationRecord.connection + @language = language_code + @resource = params.fetch('resource', 'posts') + @mode = params.fetch('mode', 'summary') + @inclusive = ['summary', 'details'].include?(@mode) + @axis = params.fetch('axis', @resource == 'posts' ? 'tags' : 'month') + @kind = params.fetch('kind', 'internal') + @grain = params.fetch('grain', 'month') + raise ArgumentError unless ['posts', 'tags'].include?(@resource) + raise ArgumentError unless ['summary', 'details', 'state', 'trend', 'snapshot'].include?(@mode) + raise ArgumentError unless ['internal', 'external', 'all'].include?(@kind) + raise ArgumentError unless ['day', 'month', 'year'].include?(@grain) + + raise ArgumentError unless ['tags', 'month', 'user'].include?(@axis) + if @mode == 'summary' && @axis == 'tags' && @resource != 'posts' + raise ArgumentError + end + + @page = integer('page', 1) + @limit = integer('limit', 20) + @tag = integer('tag') if params['tag'].present? + raise ArgumentError if @limit > 100 || @page > 1_000_000 + raise ArgumentError if @tag && @kind == 'all' + raise ArgumentError if @mode == 'trend' && !@tag + + @now = Time.current.utc + @at = timestamp('at') || @now + raise ArgumentError if @at > @now + end + + def call(metadata: true) + @metadata = metadata + # Keep totals, rows, and detail pages on one repeatable-read database snapshot. + ApplicationRecord.transaction do + unless @history_loaded + @post_start = HistoryAccess.visible(PostVersion.all).minimum(:created_at) + @tag_start = HistoryAccess.visible(TagVersion.all).minimum(:created_at) + @history_loaded = true + end + result = { population_count: HistoryAccess.population_count( + @resource == 'posts' ? PostVersion : TagVersion), page: @page, limit: @limit, at: @at.iso8601(6), + coverage: { basis: 'recorded_snapshots', exact: false, + post_history_from: @post_start&.iso8601(6), + tag_history_from: @tag_start&.iso8601(6) } } + case @mode + when 'trend' + result.merge(trend) + when 'summary' + result.merge(metadata ? { totals: totals } : {}).merge(summary) + when 'details', 'snapshot' + result.merge(metadata ? { totals: totals } : {}).merge(details) + else + result.merge(metadata ? { totals: totals } : {}).merge(listing) + end + end + end + + def each_page + return enum_for(:each_page) unless block_given? + + @page = 1 + first = call + yield first + total = ['summary', 'state', 'trend'].include?(@mode) ? first[:group_count] : first[:count] + while @page * @limit < total + @page += 1 + yield call(metadata: false) + end + end + + def validate_query! + filtered_scope(@resource) + case @mode + when 'summary' + if @axis == 'tags' + order_sql({ 'name' => 'name', 'count' => 'post_count', 'tag_id' => 'tag_id', + 'kind' => 'kind', 'deprecated' => 'deprecated_at' }, 'count:desc', []) + else + record_query('summary') + end + when 'details', 'snapshot' + record_query('details') + end + self + end + + private + + def quote(value) = @connection.quote(value) + + def integer(key, default = nil) + value = @params[key] + return default if value.blank? && default + raise ArgumentError unless value&.match?(/\A[1-9][0-9]{0,9}\z/) + + value.to_i + end + + def timestamp(key) + value = @params[key].presence + return nil unless value + pattern = /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d{1,6})?)?Z\z/ + raise ArgumentError unless value.match?(pattern) + + Date.iso8601(value.first(10)) + Time.iso8601(value.length == 17 ? value.sub(/Z\z/, ':00Z') : value).utc + end + + # Version numbers order equal-timestamp writes. Do not filter tags before this selection. + def latest_condition(table, row, boundary) + key = table == 'post_versions' ? 'post_id' : 'tag_id' + <<~SQL.squish + #{ row }.created_at #{ @inclusive ? '<=' : '<' } #{ boundary } AND NOT EXISTS ( + SELECT 1 FROM #{ table } newer + WHERE newer.#{ key } = #{ row }.#{ key } + AND newer.created_at #{ @inclusive ? '<=' : '<' } #{ boundary } + AND newer.version_no > #{ row }.version_no) + SQL + end + + def state_sql(table, boundary, live: false) + model = { 'post_versions' => PostVersion, 'tag_versions' => TagVersion, + 'nico_tag_versions' => NicoTagVersion }.fetch(table) + sql = "SELECT v.* FROM #{ table } v WHERE #{ latest_condition(table, 'v', boundary) } " + + "AND (#{ HistoryAccess.sql(model, 'v') })" + live ? "#{ sql } AND v.event_type <> 'discard'" : sql + end + + def membership(row) + key = @kind == 'external' ? 'external_tag_id' : 'tag_id' + "JSON_CONTAINS(#{ row }.tags_json, JSON_OBJECT('#{ key }', #{ @tag }))" + end + + # Deprecation keeps a tag in existence. Only an explicit discard removes it. + def tag_visible(boundary) + return '1 = 1' if @kind == 'external' + + <<~SQL.squish + NOT EXISTS (SELECT 1 FROM tag_versions t + WHERE t.tag_id = #{ @tag } AND t.event_type = 'discard' + AND #{ latest_condition('tag_versions', 't', boundary) }) + SQL + end + + def ctes + boundary = quote(@at) + <<~SQL + WITH post_at AS (#{ filtered_scope('posts').to_sql }), + tag_at AS (#{ state_sql('tag_versions', boundary) }), + nico_at AS (#{ state_sql('nico_tag_versions', boundary) }), + pairs AS ( + SELECT DISTINCT p.post_id, + CASE WHEN entry.tag_id IS NULL THEN 'external' ELSE 'internal' END AS kind, + COALESCE(entry.tag_id, entry.external_tag_id) AS tag_id + FROM post_at p + JOIN JSON_TABLE(CAST(p.tags_json AS JSON), '$[*]' COLUMNS ( + tag_id BIGINT PATH '$.tag_id' NULL ON EMPTY ERROR ON ERROR, + external_tag_id BIGINT PATH '$.external_tag_id' NULL ON EMPTY ERROR ON ERROR + )) entry + LEFT JOIN tag_at t ON t.tag_id = entry.tag_id + WHERE COALESCE(entry.tag_id, entry.external_tag_id) IS NOT NULL + AND (t.event_type IS NULL OR t.event_type <> 'discard') + ) + SQL + end + + def totals + row = @connection.select_one(<<~SQL) + #{ ctes } + SELECT (SELECT COUNT(*) FROM post_at) AS posts, + (SELECT COUNT(*) FROM (#{ filtered_scope('tags').to_sql }) counted_tags) AS tags, + (SELECT COUNT(*) FROM tag_at WHERE event_type <> 'discard' + AND deprecated_at IS NOT NULL) AS deprecated_tags, + (SELECT COUNT(*) FROM pairs) AS assignments, + (SELECT COUNT(*) FROM pairs WHERE kind = 'external') AS external_assignments + SQL + row.transform_values!(&:to_i) + row.merge('posts' => available?(@post_start, @at) ? row['posts'] : nil, + 'assignments' => available?(@post_start, @at) ? row['assignments'] : nil, + 'external_assignments' => available?(@post_start, @at) ? + row['external_assignments'] : nil, + 'tags' => available?(@tag_start, @at) ? row['tags'] : nil, + 'deprecated_tags' => available?(@tag_start, @at) ? row['deprecated_tags'] : nil) + end + + def available?(first, boundary) + first && (@inclusive ? first <= boundary : first < boundary) + end + + def listing + catalog = <<~SQL + , tag_counts AS ( + SELECT membership.kind, membership.tag_id, COUNT(*) AS post_count + FROM pairs membership GROUP BY membership.kind, membership.tag_id + ), catalog AS ( + SELECT 'internal' AS kind, tag_id, name, names, + deprecated_at FROM tag_at WHERE event_type <> 'discard' + UNION ALL + SELECT tc.kind, tc.tag_id, external_version.name, JSON_OBJECT() AS names, + NULL AS deprecated_at + FROM tag_counts tc + LEFT JOIN tag_at internal_version + ON tc.kind = 'internal' AND internal_version.tag_id = tc.tag_id + LEFT JOIN nico_at external_version + ON tc.kind = 'external' AND external_version.tag_id = tc.tag_id + WHERE tc.kind = 'external' OR internal_version.tag_id IS NULL + ) + SQL + filters = [] + filters << "c.kind = #{ quote(@kind) }" unless @kind == 'all' + filters << "c.tag_id = #{ @tag }" if @tag + where = filters.any? ? "WHERE #{ filters.join(' AND ') }" : '' + source = 'FROM catalog c LEFT JOIN tag_counts n ON n.kind = c.kind AND n.tag_id = c.tag_id' + prefix = "#{ ctes } #{ catalog }" + total = if @metadata + @connection.select_value("#{ prefix } SELECT COUNT(*) #{ source } #{ where }").to_i + end + order = order_sql({ 'name' => 'c.name', 'count' => 'COALESCE(n.post_count, 0)', + 'tag_id' => 'c.tag_id', 'kind' => 'c.kind', + 'deprecated' => 'c.deprecated_at' }, 'count:desc', + ['c.kind ASC', 'c.tag_id ASC']) + rows = @connection.select_all(<<~SQL) + #{ prefix } + SELECT c.kind, c.tag_id, c.name, c.names, c.deprecated_at, + COALESCE(n.post_count, 0) AS post_count #{ source } #{ where } + ORDER BY #{ order } LIMIT #{ @limit } OFFSET #{ (@page - 1) * @limit } + SQL + { count: total, group_count: total, state_rows: rows.map do |row| + { kind: row['kind'], tag_id: row['tag_id'], name: row['name'], + names: row['names'].is_a?(String) ? JSON.parse(row['names']) : row['names'], + deprecated: row['deprecated_at']&.iso8601, + count: available?(@post_start, @at) ? row['post_count'].to_i : nil, + at: @at.iso8601(6) } + end } + end + + # All record filters apply after selecting the latest recorded version at the point. + def restored_scope(resource) + model = resource == 'posts' ? PostVersion : TagVersion + table = model.table_name + scope = model.where(Arel.sql(latest_condition(table, table, quote(@at)))) + .where.not(event_type: 'discard') + if @tag + scope = if resource == 'posts' + scope.where("#{ membership(table) } AND #{ tag_visible(quote(@at)) }") + elsif @kind == 'internal' + scope.where(tag_id: @tag) + else + scope.none + end + end + scope + end + + def filtered_scope(resource) + params = @params.slice('from', 'before', 'user_id', 'resource_id') + .merge('mode' => 'details', 'latest' => '0', 'sort' => '') + CountsQuery.new(restored_scope(resource), resource, params, + latest_expression: '1').relation + end + + def record_query(mode) + params = @params.slice('locale', 'sort', 'page', 'limit') + .merge('mode' => mode, 'latest' => '0', 'axis' => @axis == 'tags' ? 'month' : @axis, + 'page' => @page.to_s, 'limit' => @limit.to_s) + CountsQuery.new(filtered_scope(@resource), @resource, params, latest_expression: '1') + end + + def summary + return listing if @axis == 'tags' + + query = record_query('summary') + { count: @metadata ? query.count : nil }.merge(query.summary(include_count: @metadata)) + end + + def details + query = record_query('details') + { count: @metadata ? query.count : nil, + versions: CountsVersionRepr.many(query.details, resource: @resource, + language_code: @language, at: @at, + inclusive: @inclusive) } + end + + def next_boundary(time) + case @grain + when 'day' then time.beginning_of_day + 1.day + when 'month' then time.beginning_of_month.next_month + else time.beginning_of_year.next_year + end + end + + def periods + @before = timestamp('before') || @now.beginning_of_month + @from = timestamp('from') || @before.prev_year + raise ArgumentError if @from >= @before || @before > @now + + rows = [] + boundary = next_boundary(@from) + while boundary <= @before + raise ArgumentError if rows.length >= MAX_PERIODS + + format = { 'day' => '%Y-%m-%d', 'month' => '%Y-%m', 'year' => '%Y' }.fetch(@grain) + rows << [boundary, (boundary - 1.second).strftime(format)] + boundary = next_boundary(boundary) + end + rows + end + + def trend + points = periods + metadata = { count: points.length, group_count: points.length, state_rows: [], + from: @from.iso8601, before: @before.iso8601, grain: @grain } + return metadata if points.empty? + + values = points.map do |boundary, period| + "SELECT CAST(#{ quote(boundary) } AS DATETIME(6)) AS boundary, " + + "#{ quote(period) } AS period" + end.join(' UNION ALL ') + order = order_sql({ 'period' => 'boundary', 'count' => 'post_count' }, + 'period:asc', ['boundary ASC']) + known = @post_start ? "periods.boundary > #{ quote(@post_start) }" : 'FALSE' + rows = @connection.select_all(<<~SQL) + WITH periods AS (#{ values }), series AS ( + SELECT periods.boundary, periods.period, + CASE WHEN #{ known } THEN COUNT(DISTINCT p.post_id) ELSE NULL END AS post_count + FROM periods + LEFT JOIN post_versions p ON p.event_type <> 'discard' + AND #{ membership('p') } + AND #{ latest_condition('post_versions', 'p', 'periods.boundary') } + AND #{ tag_visible('periods.boundary') } + GROUP BY periods.boundary, periods.period + ) + SELECT * FROM series ORDER BY #{ order } + LIMIT #{ @limit } OFFSET #{ (@page - 1) * @limit } + SQL + metadata.merge(state_rows: rows.map do |row| + { kind: @kind, tag_id: @tag, name: nil, period: row['period'], + at: row['boundary'].utc.iso8601(6), count: row['post_count']&.to_i } + end) + end + + def order_sql(expressions, default, final) + raw = @params.fetch('sort', default) + sorts = raw.empty? ? [] : QueryState.list(raw).map { _1.split(':', -1) } + raise ArgumentError if sorts.map(&:first).uniq.length != sorts.length + raise ArgumentError if sorts.length > expressions.length + + order = sorts.flat_map do |key, direction, *extra| + if extra.any? || !expressions.key?(key) || !['asc', 'desc'].include?(direction) + raise ArgumentError + end + expression = expressions.fetch(key) + ["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"] + end + (order + final).join(', ') + end +end diff --git a/backend/app/services/counts_table_export.rb b/backend/app/services/counts_table_export.rb new file mode 100644 index 0000000..eab1d50 --- /dev/null +++ b/backend/app/services/counts_table_export.rb @@ -0,0 +1,350 @@ +require 'csv' + +# Export uses the exact same population and grouped cells as the interactive table. +class CountsTableExport + EXTRA = ['format', 'columns', 'headers', 'labels', 'children', 'child_sort', + 'order', 'nested_json', 'summary_columns', 'basis', 'axis', 'latest'].freeze + attr_reader :format, :resource + + def initialize(params) + params = CountsResourceDefinition.normalise_params(params) + QueryState.validate!(params, CountsTableQuery::KEYS + EXTRA + ['measures']) + raise ArgumentError unless params.values.all? { _1.is_a?(String) } + + params = CountsDefaults.for(params).merge(params) + unless params.key?('children') + params = params.merge('children' => CountsExportProjection.default_children( + params.fetch('resource', 'posts')).join(' ')) + end + @params = params + @labels = JSON.parse(params.fetch('labels', '{}')) + raise ArgumentError unless @labels.is_a?(Hash) + @selected = params.key?('children') ? QueryState.list(params['children']) : nil + @order = QueryState.list(params.fetch('order', '')) + @child_sort = QueryState.list(params.fetch('child_sort', '')).map { _1.split(':', -1) } + @child_sort.each do |path, direction, *extra| + raise ArgumentError if extra.any? || !['asc', 'desc'].include?(direction) + raise ArgumentError unless path&.match?(/\A[a-zA-Z_]+(?:\.[a-zA-Z_]+)*\/[a-zA-Z_]+\z/) + end + @format = params.fetch('format', 'json') + @resource = params.fetch('resource', 'posts') + raise ArgumentError unless ['json', 'csv'].include?(format) + + @query = CountsTableQuery.new(params.select { |key, _value| CountsTableQuery::KEYS.include?(key) || + key == 'measures' || QueryState.valid_key?(key) } + .merge('page' => '1', 'limit' => '100')) + @summary = @query.mode == 'summary' + default_columns = @summary ? @query.axes.all.map { _1.fetch('field') } + ['count'] : + CountsExportProjection.default_columns(resource) + @columns = QueryState.list(params.fetch('columns', default_columns.join(' '))) + default_headers = @columns.map { @labels.fetch(_1, _1) } + @headers = JSON.parse(params.fetch('headers', default_headers.to_json)) + raise ArgumentError unless @headers.is_a?(Array) && @headers.all? { _1.is_a?(String) } + if @summary && @headers.any? && @headers.length != @query.axes.all.length + 1 + raise ArgumentError + end + + unless @summary + if ['posts', 'tags'].include?(resource) + @projection = CountsExportProjection.new(params, resource:, summary: false, tags: false) + else + allowed = @query.population.definition.fields.keys + ['latest'] + raise ArgumentError if (@columns - allowed).any? + raise ArgumentError unless @columns.length == @headers.length && @columns.uniq == @columns + end + end + end + + def body(progress: nil, phase: nil) + Enumerator.new do |output| + CountsExporter.with_read_snapshot do + processed = 0 + started = false + first = true + @query.each_page(before_page: -> { phase&.call('aggregating') }) do |page| + total = @summary ? nil : page[:count] + progress&.call(processed, total) + phase&.call('writing') + unless started + metadata = page.except(:versions, :cells, :axis_labels, :axis_names, :column_values, + :row_headers, :column_headers, :pivot_totals).merge( + resource:, record_unit: @summary && format == 'json' ? 'pivot_cell' : 'row', + body_row_totals: 'once_per_logical_group', page_grand_total_repeated: false, + logical_count_unit: 'regular_result_rows', + name_collection: 'all_recorded_languages', + name_columns: @query.population.definition.name_columns, + aggregate_records_in_logical_count: false, + filters: @params.select { |key, _value| CountsTableQuery::KEYS.include?(key) || + key == 'measures' || QueryState.valid_key?(key) }, + fields: @query.population.definition.metadata, + columns: @columns, column_labels: @headers, + order: @params['order'], children: @params['children'], + sort: @params.fetch('sort', ''), child_sort: @params['child_sort']) + output << (format == 'json' ? "{\"metadata\":#{ metadata.to_json },\"data\":[" : "\uFEFF") + if format == 'csv' + headings = @summary ? pivot_headers(page) : [@headers] + headings.each do |headers| + output << CSV.generate_line(headers.map { safe(_1) }, row_sep: "\r\n") + end + end + started = true + end + if @summary + pivot_rows(page).each do |header| + records = page.fetch(:column_headers).map { pivot_cell(header, _1, page) } + if format == 'csv' + labels = pivot_row_labels(header, page) + values = labels + records.map { _1[:count] } + output << CSV.generate_line(values.map { safe(_1) }, row_sep: "\r\n") + processed += 1 + progress&.call(processed, nil) + else + records.each do |record| + output << ',' unless first + output << record.to_json + first = false + processed += 1 + progress&.call(processed, nil) + end + end + end + next + end + page[:versions].each do |row| + record = project(row) + if format == 'json' + output << ',' unless first + output << record.to_json + else + values = if @summary + row[:values].each_with_index.map do |value, index| + summary_value(value, index, page) + end + [row[:count]] + elsif @projection + @projection.csv_record(record, row) + else + record.map { |key, value| csv_logical(key, value) } + end + output << CSV.generate_line(values.map { safe(_1) }, row_sep: "\r\n") + end + first = false + processed += 1 + progress&.call(processed, total) + end + end + progress&.call(processed, processed) + output << ']}' if format == 'json' + end + end + end + + private + + def pivot_rows(page) + page.fetch(:row_headers).reject do |header| + header[:kind] == 'grand_total' && @query.page * @query.limit < page[:group_count] + end + end + + def total_label(header) + key = header[:label_key] || (header[:kind] == 'grand_total' ? + (@query.axes.all.length == 1 ? 'singleTotal' : 'grandTotal') : 'subtotal') + @labels.fetch(key, '') + end + + def pivot_row_labels(header, page) + values = header[:values].each_with_index.map { |value, index| + summary_value(value, index, page) + } + if @previous_pivot_row + values.each_index do |index| + if (@previous_pivot_row[:values].length > index && + header[:values].first(index + 1) == @previous_pivot_row[:values].first(index + 1)) + values[index] = '' + end + end + end + @previous_pivot_row = header + values << total_label(header) unless header[:kind] == 'data' + leading = [@query.axes.rows.length, @query.axes.columns.any? ? 1 : 0].max + Array.new(leading) { values[_1] || '' } + end + + def pivot_headers(page) + row_count = @query.axes.rows.length + cols = @query.axes.columns + leading = [row_count, cols.any? ? 1 : 0].max + columns = page.fetch(:column_headers) + headings = Array.new([cols.length, 1].max) do |level| + prefix = Array.new(leading) { level.zero? ? @headers[_1].to_s : '' } + prefix = [''] if row_count.zero? && cols.any? + prefix + columns.each_with_index.map do |header, index| + values = header[:values] + if cols.empty? + @labels.fetch('count', '') + elsif level == values.length && header[:kind] != 'data' + total_label(header) + elsif level < values.length + previous = index.positive? ? columns[index - 1][:values] : nil + if level < cols.length - 1 && previous && previous.first(level + 1) == values.first(level + 1) + '' + else + summary_value(values[level], row_count + level, page) + end + else + '' + end + end + end + return headings if cols.empty? + + cols.each_index.flat_map do |level| + prefix = Array.new(leading, '') + prefix = headings.first.first(leading) if level.zero? + names = columns.each_with_index.map do |header, index| + next '' if header[:values].length < level + + previous = index.positive? ? columns[index - 1][:values] : nil + if (previous && previous.length >= level && + previous.first(level) == header[:values].first(level)) + '' + else + @headers[row_count + level].to_s + end + end + headings[level] = Array.new(leading, '') + headings[level].drop(leading) if level.zero? + [prefix + names, headings[level]] + end + end + + def pivot_cell(row, column, page) + @pivot_lookup ||= {} + # Discard the previous page's lookup; memory remains bounded by one page and its rollups. + unless @pivot_page.equal?(page) + @pivot_page = page + @pivot_lookup = page.fetch(:cells).to_h do |cell| + values = cell[:values] + [[values.first(@query.axes.rows.length), values.drop(@query.axes.rows.length)], cell[:count]] + end + page.fetch(:pivot_totals).each { @pivot_lookup[[_1[:row], _1[:column]]] = _1[:count] } + end + kinds = [row[:kind], column[:kind]] + kind = if kinds.include?('grand_total') + 'grand_total' + elsif kinds.include?('subtotal') + 'subtotal' + else + 'data' + end + values = row[:values].each_with_index.to_h { |value, index| ["a#{ index }", value] } + column[:values].each_with_index do |value, index| + values["a#{ @query.axes.rows.length + index }"] = value + end + names = page.fetch(:axis_names, {}).to_h do |key, entries| + [key, entries.fetch(values[key].to_s, {})] + end + { row: row[:values], column: column[:values], kind:, + axis_names: names, + row_kind: row[:kind], column_kind: column[:kind], + row_depth: row[:values].length, column_depth: column[:values].length, + label_key: CountsPivotLayout.label_key(@query.axes.rows.length, @query.axes.columns.length, + row[:values].length, column[:values].length), + count: @pivot_lookup.fetch([row[:values], column[:values]], 0) } + end + + def summary_value(value, index, page) + return '********' if value == CountsAxes::MASKED_VALUE + + field = @query.axes.all.fetch(index).fetch('field') + return @labels.fetch('bot', '') if field == 'user' && value.nil? + if ['language_code', 'source_language_code'].include?(field) + return @labels.fetch("language:#{ value }", value) + end + + names = page[:axis_labels].fetch("a#{ index }", {}) + name = field == 'user' ? names[value.to_s] : names.fetch(value.to_s, value) + if field == 'user' + name = @labels.fetch('notRegistered', '') if name.nil? || name == '' + pattern = @labels.fetch('actorFormat', '{{name}} (#{{id}})') + return pattern.gsub(/\{\{(name|id)\}\}/) do + Regexp.last_match(1) == 'name' ? name.to_s : value.to_s + end + end + @labels.fetch("#{ field }:#{ value }", name) + end + + def project(row) + return { row: row[:values].first(@query.axes.rows.length), + column: row[:values].drop(@query.axes.rows.length), count: row[:count] } if @summary + return @projection.project(row) if @projection + + @columns.to_h do |column| + key = column == 'user' ? 'created_by_user' : column + value = row[key] + value = project_nested(value, key) if ['members', 'placements', 'export_paths_json'].include?(key) + [key, value] + end + end + + def camel(key) + key.gsub(/_([a-z])/) { Regexp.last_match(1).upcase } + end + + def project_nested(value, path) + rows = path == 'export_paths_json' ? (value || {}).map { |profile, target| + { 'profile' => profile, 'path' => target } + } : Array(value) + sorts = @child_sort.filter_map do |key, direction| + parent, field = key.split('/', 2) + [field, direction] if parent == path + end + rows = rows.each_with_index.sort do |(first, a), (last, b)| + compared = 0 + sorts.each do |field, direction| + left = first.find { |key, _| camel(key) == field }&.last + right = last.find { |key, _| camel(key) == field }&.last + compared = if left.nil? || right.nil? + left.nil? == right.nil? ? 0 : (left.nil? ? 1 : -1) + else + (left.to_s <=> right.to_s) * (direction == 'asc' ? 1 : -1) + end + break unless compared.zero? + end + compared.zero? ? a <=> b : compared + end.map(&:first) + rows.map do |row| + fields = row.keys.select { !@selected || @selected.include?("#{ path }.#{ camel(_1) }") } + ordered = (@order + fields.map { "#{ path }.#{ camel(_1) }" }).uniq + .filter_map { |item| fields.find { "#{ path }.#{ camel(_1) }" == item } } + ordered.to_h do |key| + item = row[key] + item = project_nested(item, "#{ path }.#{ camel(key) }") if item.is_a?(Array) + [key, item] + end + end + end + + def csv_logical(key, value) + return @labels.fetch('bot', '') if key == 'created_by_user' && value.nil? + return value ? @labels.fetch('latest', '') : '' if key == 'latest' + if value.is_a?(Time) || value.is_a?(ActiveSupport::TimeWithZone) + return @query.population.timezone.utc_to_local(value.utc).iso8601 + end + if key == 'language_code' || key == 'source_language_code' + return @labels.fetch("language:#{ value }", value) + end + if value.is_a?(Hash) + return value.to_h { |field, item| [@labels.fetch(camel(field.to_s), ''), csv_logical(field, item)] } + end + return value.map { csv_logical(key, _1) } if value.is_a?(Array) + + @labels.fetch("#{ key }:#{ value }", value) + end + + def safe(value) + return value if value.is_a?(Numeric) + + text = value.is_a?(Hash) || value.is_a?(Array) ? JSON.generate(value) : value.to_s + text.match?(/\A\s*[=+@-]/) ? "'#{ text }" : text + end +end diff --git a/backend/app/services/counts_table_query.rb b/backend/app/services/counts_table_query.rb new file mode 100644 index 0000000..92acf8c --- /dev/null +++ b/backend/app/services/counts_table_query.rb @@ -0,0 +1,528 @@ +# Common entrypoint for six version resources; legacy endpoints keep their contracts. +class CountsTableQuery + SQL_TIMEOUT_SECONDS = 30 + SQL_HINTS = ["MAX_EXECUTION_TIME(#{ SQL_TIMEOUT_SECONDS * 1000 })", + "SET_VAR(optimizer_switch='semijoin=off')"].freeze + MAX_COLUMNS = 100 + class TooManyColumns < ArgumentError; end + + KEYS = ['resource', 'definition', 'mode', 'at', 'execution_at', 'from', 'through', + 'timezone', 'rows', 'cols', 'conditions', 'drill', 'tags', 'match', + 'tag', 'kind', 'resource_id', 'user_id', 'page', 'limit', 'sort', 'locale', + 'measures', 'column_order', 'condition_values', 'name_scope'].freeze + attr_reader :population, :axes, :page, :limit, :mode + + def initialize(params) + params = CountsResourceDefinition.normalise_params(params) + QueryState.validate!(params, KEYS + ['measures']) + raise ArgumentError unless params.values.all? { _1.is_a?(String) } + + params = CountsDefaults.for(params).slice(*KEYS).merge(params) + @params = params + @mode = params.fetch('mode', 'summary') + validate_summary_conditions! + @population = CountsPopulation.new(params, authorisation_flg: mode != 'summary') + @definition = population.definition + @connection = @definition.model.connection + @axes = CountsAxes.new(population) + @page = Integer(params.fetch('page', '1'), 10) + @limit = Integer(params.fetch('limit', '100'), 10) + @mode = params.fetch('mode', 'summary') + raise ArgumentError unless page.between?(1, 1_000_000) && limit.between?(1, 100) + raise ArgumentError unless ['summary', 'details'].include?(mode) + + @sort = QueryState.list(params.fetch('sort', '')).map { _1.split(':', -1) } + allowed = mode == 'summary' ? axes.all.each_index.map { "a#{ _1 }" } + ['count'] : + @definition.fields.keys + ['latest'] + + (@definition.resource == 'tags' ? @definition.name_columns.map { _1[:key] } : []) + + (@definition.resource == 'posts' ? ['tags', 'parents'] : []) + + (@definition.resource == 'tags' ? ['parents', 'deprecated'] : []) + raise ArgumentError if @sort.map(&:first).uniq.length != @sort.length + @sort.each do |key, direction, *extra| + measure = mode == 'summary' && key&.match?(/\Am[0-9]{1,2}\z/) + measure_values(key) if measure + if extra.any? || (!allowed.include?(key) && !measure) || !['asc', 'desc'].include?(direction) + raise ArgumentError + end + end + end + + def call + ApplicationRecord.transaction do + result = metadata.merge(mode == 'summary' ? summary : details) + visible = mode == 'summary' ? result[:visible_output_count] : result[:count] + totals = output_totals(visible) + response = result.merge(visible_output_count: visible, + **totals, + output_count_scope: 'before_authorisation', + output_count_unit: 'regular_result_rows', + page_grand_total_repeated: mode == 'summary' && axes.rows.any? && + result[:group_count] > limit) + if mode == 'summary' + response.except!(:excluded_count, :excluded_count_scope) + response[:column_values_editable] = HistoryAccess.admin? || + !result.fetch(:column_values, []).flatten.include?(CountsAxes::MASKED_VALUE) + end + response + end + end + + def metadata + { condition_labels: condition_labels, population_count: nil, population_count_computed: false, + excluded_count: nil, excluded_count_scope: 'current_query', + population_count_scope: 'all_recorded_versions', + page:, limit:, execution_at: population.at.iso8601(6), at: @params['at'].presence, + definition: population.history ? 'history' : 'latest', + timezone: population.timezone.identifier, rows: axes.rows, cols: axes.columns, + fields: @definition.metadata, + masked_value: HistoryAccess.admin? ? nil : CountsAxes::MASKED_VALUE, + count: record_counts[:visible] } + end + + def summary + if axes.all.empty? + headers = [{ values: [], kind: 'data', depth: 0, axis_count: 0, label_key: nil }] + return { group_count: 1, visible_output_count: 1, column_values: [[]], + row_headers: page == 1 ? headers : [], column_headers: headers, pivot_totals: [], + cells: page == 1 ? [{ values: [], count: record_counts[:visible] }] : [], + axis_labels: {} } + end + source = summary_population.scope + source = source.joins(axes.joins) unless axes.joins.empty? + sql = axes.grouped_sql(source) + prefix = "WITH cells AS (#{ sql })" + column_values = limited_columns(prefix) + row_keys = axes.rows.each_index.map { "a#{ _1 }" } + # Page whole row tuples so a pivot row is never split across display pages. + row_selection = row_keys.empty? ? '1 AS singleton' : row_keys.join(', ') + row_group = row_keys.empty? ? 'GROUP BY singleton' : "GROUP BY #{ row_keys.join(', ') }" + measure_joins = [] + extras = [] + row_sort = @sort.each_with_index.filter_map do |(key, direction), index| + if key == 'count' || key.match?(/\Am[0-9]+\z/) + values = key == 'count' ? [] : measure_values(key) + relation = measure_relation(values) + matches = row_keys.map { "grouped.#{ _1 } <=> sort#{ index }.#{ _1 }" } + condition = matches.empty? ? 'TRUE' : matches.join(' AND ') + measure_joins << "LEFT JOIN (#{ relation }) sort#{ index } ON #{ condition }" + extras << "sort#{ index }.count AS sort#{ index }" + "sort#{ index } IS NULL ASC, sort#{ index } #{ direction.upcase }" + elsif row_keys.include?(key) + axis_order_sql(key, direction) + end + end + order_rows = row_sort + row_keys.map { axis_order_sql(_1) } + order_rows = ['singleton ASC'] if order_rows.empty? + grouped = "SELECT #{ row_selection } FROM cells #{ row_group }" + selection = (['grouped.*'] + extras).join(', ') + group_sql = "SELECT #{ selection } FROM (#{ grouped }) grouped #{ measure_joins.join(' ') }" + join = row_keys.map { "cells.#{ _1 } <=> page_rows.#{ _1 }" }.join(' AND ') + join = 'TRUE' if row_keys.empty? + order_cols = @sort.filter_map do |key, direction| + next unless key.match?(/\Aa[0-9]+\z/) && !row_keys.include?(key) + + axis_order_sql(key, direction, qualifier: 'cells') + end + order = ['page_rows.position'] + order_cols + + axes.all.each_index.map { axis_order_sql("a#{ _1 }", qualifier: 'cells') } + result = @connection.select_all(<<~SQL).to_a + #{ prefix }, row_groups AS (#{ group_sql }), + numbered AS (SELECT row_groups.*, + ROW_NUMBER() OVER (ORDER BY #{ order_rows.join(', ') }) AS position FROM row_groups), + group_totals AS (SELECT COUNT(*) AS group_count FROM row_groups) + SELECT /*+ #{ SQL_HINTS.join(' ') } */ cells.*, + page_rows.position AS counts_position, group_totals.group_count AS counts_group_count + FROM group_totals LEFT JOIN numbered page_rows + ON page_rows.position > #{ (page - 1) * limit } + AND page_rows.position <= #{ page * limit + 1 } + LEFT JOIN cells ON #{ join } + ORDER BY #{ order.join(', ') } + SQL + total = result.first.fetch('counts_group_count').to_i + next_row = result.find { _1['counts_position'].to_i == page * limit + 1 } + records = result.select do |record| + record['counts_position'] && record['counts_position'].to_i <= page * limit + end + row_values = records.map { |record| row_keys.map { record[_1] } }.uniq + row_depths = CountsPivotLayout.depths(row_keys, @sort) + column_keys = axes.columns.each_index.map { "a#{ axes.rows.length + _1 }" } + column_sort = @sort + if @params.key?('column_order') && column_keys.any? && + !(@sort.first&.first == 'count' || @sort.first&.first&.match?(/\Am[0-9]+\z/)) + column_sort = [[column_keys.first, 'asc']] + @sort + end + column_depths = CountsPivotLayout.depths(column_keys, column_sort, measures: false) + row_headers = CountsPivotLayout.headers(row_values, row_depths, + next_values: next_row && row_keys.map { next_row[_1] }, grand: axes.rows.any?, + axis_count: axes.rows.length, other_count: axes.columns.length) + if axes.rows.empty? && axes.columns.any? + row_headers = CountsPivotLayout.headers([[]], [], axis_count: 0, + other_count: axes.columns.length).map { _1.merge(kind: 'grand_total') } + end + column_headers = CountsPivotLayout.headers(column_values, column_depths, + grand: axes.columns.any?, axis_count: axes.columns.length, + other_count: axes.rows.length, direction: :column) + if axes.columns.empty? && axes.rows.any? + column_headers = CountsPivotLayout.headers([[]], [], axis_count: 0, + other_count: axes.rows.length, direction: :column) + .map { _1.merge(kind: 'grand_total') } + end + totals = CountsPivotTotals.new(summary_population, axes, row_headers, column_headers).call + { group_count: total, + visible_output_count: axes.rows.empty? ? 1 : total, + column_values:, row_headers:, column_headers:, pivot_totals: totals, + cells: records.map do |row| + { values: axes.all.each_index.map { row["a#{ _1 }"] }, count: row['count'].to_i } + end, axis_labels: axis_labels(records, column_values), axis_names: @axis_names } + end + + def details + query = population.scope.optimizer_hints(*SQL_HINTS).select(Arel.sql( + "(#{ population.current_latest_sql }) AS counts_latest")) + order = sort_sql { detail_expression(_1) } + query = query.order(Arel.sql((order + ['v.id ASC']).join(', '))) + .limit(limit).offset((page - 1) * limit) + rows = query.to_a + if ['posts', 'tags'].include?(@definition.resource) + point = population.history ? population.at : + population.timestamp(@params['at']) || population.at + values = CountsVersionRepr.many(rows, resource: @definition.resource, + language_code: locale.language_code, + at: point, history_flg: population.history) + values.each_with_index { |value, index| value[:event_type] = rows[index].event_type } + else + users = User.where(id: rows.map(&:created_by_user_id)).pluck(:id, :name).to_h + values = rows.map do |row| + attributes = @definition.fields.keys.to_h do |key| + [key, row.public_send(@definition.column(key))] + end + attributes.merge('id' => row.id, 'resource_id' => row[@definition.keys.first], + 'latest' => row.attributes['counts_latest'].to_i == 1, + 'created_by_user' => row.created_by_user_id && + { id: row.created_by_user_id, name: users[row.created_by_user_id] }) + end + end + { versions: values } + end + + def each_page(before_page: nil) + total = nil + loop do + before_page&.call + result = call + total ||= mode == 'summary' ? result[:group_count] : result[:count] + yield result + break if page * limit >= total + + @page += 1 + end + end + + private + + # Public timeline predicates may include hidden records. Private predicates must + # never become a count oracle, even when the selected axes themselves are public. + def validate_summary_conditions! + return if mode != 'summary' || HistoryAccess.admin? + + fields = QueryState.records(@params, 'conditions') + QueryState.records(@params, 'drill') + private_fields = fields.any? { !CountsAxes::PUBLIC_FIELDS.include?(_1['field']) } + keys = ['tags', 'tag', 'kind', 'resource_id', 'user_id'] + private_parameters = keys.any? do |key| + @params.any? { |name, value| (name == key || name.start_with?("#{ key }.")) && value != '' } + end + raise CountsAxes::PrivateCondition if private_fields || private_parameters + + ['column_order', 'measures'].each do |key| + # Coordinates are compared only to already masked expressions, never raw + # values. Public coordinates remain sortable; a guessed private ID cannot match. + raise CountsAxes::PrivateCondition if QueryState.records(@params, key).any? do |record| + record['value'] == CountsAxes::MASKED_VALUE + end + end + end + + def unrestricted_population + @unrestricted_population ||= CountsPopulation.new( + @params.merge('execution_at' => population.at.iso8601(6)), authorisation_flg: false) + end + + def summary_population + population + end + + def record_counts + return @record_counts if @record_counts + + if HistoryAccess.admin? || mode == 'summary' + total = population.scope.except(:select) + .optimizer_hints(*SQL_HINTS).count + return @record_counts = { total:, visible: total, denied: 0 } + end + total, denied = unrestricted_population.scope.except(:select) + .optimizer_hints(*SQL_HINTS).pick(Arel.sql('COUNT(*)'), + Arel.sql("COALESCE(SUM(CASE WHEN #{ HistoryAccess.sql(@definition.model, 'v') } " + + 'THEN 0 ELSE 1 END), 0)')) + # Related/previous-version predicates may differ under public authority. + shared = QueryState.records(@params, 'conditions').empty? && + QueryState.records(@params, 'drill').none? { _1['field'] == 'changed_field' } + visible = if shared + total.to_i - denied.to_i + else + population.scope.except(:select) + .optimizer_hints(*SQL_HINTS).count + end + @record_counts = { total: total.to_i, denied: denied.to_i, visible: } + end + + def output_totals visible_rows + return @output_totals if @output_totals + + if mode == 'summary' + return @output_totals = { output_count: visible_rows, excluded_count: nil } + end + if HistoryAccess.admin? + return @output_totals = { output_count: visible_rows, excluded_count: 0 } + end + # Only numerical aggregates leave this population. Labels, snapshots and + # detail pagination continue to use the authorised population exclusively. + source = unrestricted_population + rows = if mode == 'details' + record_counts[:total] + else + summary_output_count(source) + end + @output_totals = { output_count: rows, excluded_count: record_counts[:denied] } + end + + def summary_output_count(source) + keys = axes.rows.each_index.map { "a#{ _1 }" } + return 1 if keys.empty? + + unrestricted_axes = CountsAxes.new(source) + relation = source.scope + joins = unrestricted_axes.joins + relation = relation.joins(joins) unless joins.empty? + expressions = unrestricted_axes.expressions + selected = expressions.each_with_index.map { |expression, index| + "#{ expression } AS a#{ index }" + } + sql = relation.except(:select).select(Arel.sql(selected.join(', '))) + .distinct.to_sql + prefix = "WITH cells AS (#{ sql })" + @connection.select_value(<<~SQL).to_i + #{ prefix } + SELECT /*+ #{ SQL_HINTS.join(' ') } */ COUNT(*) FROM ( + SELECT #{ keys.join(', ') } FROM cells GROUP BY #{ keys.join(', ') } + ) output_groups + SQL + end + + def locale = @locale ||= Locale.find_by!(code: @params.fetch('locale', 'ja')) + + def limited_columns prefix + keys = axes.columns.each_index.map { "a#{ axes.rows.length + _1 }" } + return [[]] if keys.empty? + + configured = @sort.select { |key, _direction| keys.include?(key) } + directions = (configured + keys.map { [_1, 'asc'] }).uniq { _1.first } + order = directions.map { |key, direction| axis_order_sql(key, direction) }.join(', ') + custom = QueryState.records(@params, 'column_order') + if custom.any? + raise ArgumentError unless custom.map { _1['value'] }.uniq.length == custom.length + + cases = custom.each_with_index.map do |record, index| + "WHEN #{ keys.first } <=> #{ @connection.quote(record['value']) } THEN #{ index }" + end.join(' ') + order = "CASE #{ cases } ELSE #{ custom.length } END, #{ order }" + end + values = @connection.select_rows(<<~SQL) + #{ prefix } + SELECT /*+ #{ SQL_HINTS.join(' ') } */ DISTINCT #{ keys.join(', ') } FROM cells + ORDER BY #{ order } LIMIT #{ MAX_COLUMNS + 1 } + SQL + raise TooManyColumns if values.length > MAX_COLUMNS + + values + end + + def measure_relation values + source = summary_population.scope.except(:select, :order) + # Omitted multi-value column axes still determine full-cell membership. + membership = summary_population.scope.except(:select, :order) + all_joins = axes.joins + membership = membership.joins(all_joins) unless all_joins.empty? + source = source.where("v.id IN (#{ membership.select('v.id').distinct.to_sql })") + fields = axes.rows + axes.columns.first(values.length) + joins = axes.joins(fields.map { _1.fetch('field') }) + source = source.joins(joins) unless joins.empty? + values.each_with_index do |value, index| + expression = axes.expressions[axes.rows.length + index] + source = source.where("(#{ expression }) <=> ?", value) + end + axes.grouped_sql(source, axes.rows.each_index.to_a) + end + + def axis_order_sql key, direction = 'asc', qualifier: nil + column = qualifier ? "#{ qualifier }.#{ key }" : key + expression = column + field = axes.all.fetch(key.delete_prefix('a').to_i).fetch('field') + if !HistoryAccess.admin? && [:integer, :boolean].include?(@definition.fields[field]) + masked = @connection.quote(CountsAxes::MASKED_VALUE) + expression = "CASE WHEN #{ column } <=> #{ masked } THEN NULL " + + "ELSE CAST(#{ column } AS SIGNED) END" + return "#{ column } IS NULL ASC, (#{ column } <=> #{ masked }) ASC, " + + "#{ expression } #{ direction.upcase }" + end + if field == 'event_type' + expression = event_order_sql(column) + elsif field == 'category' + # Tag's enum preserves the same category order as TagsController. + cases = Tag.categories.keys.each_with_index.map { |category, index| + "WHEN #{ @connection.quote(category) } THEN #{ index }" + }.join(' ') + expression = "CASE #{ column } #{ cases } ELSE #{ Tag.categories.length } END" + end + "#{ column } IS NULL ASC, #{ expression } #{ direction.upcase }" + end + + def measure_values key + records = QueryState.records(@params, 'measures').select { _1['key'] == key } + raise ArgumentError if records.empty? + raise ArgumentError unless records.map { _1['axis'] }.sort == (0...records.length).to_a + + value = records.sort_by { _1['axis'] }.map { _1['value'] } + unless (value.is_a?(Array) && value.length <= axes.columns.length && + value.all? { _1.nil? || _1.is_a?(String) || _1.is_a?(Numeric) }) + raise ArgumentError + end + value + end + + def detail_expression(key) + return "(#{ population.current_latest_sql })" if key == 'latest' + return event_order_sql(@definition.expression(key)) if key == 'event_type' + return @definition.name_expression(key) if @definition.resource == 'tags' && + @definition.name_column(key) + aliases = { 'tags' => 'tags_json', 'parents' => @definition.resource == 'posts' ? + 'parent_post_ids' : 'parent_tag_ids', 'deprecated' => 'deprecated_at' } + field = aliases.fetch(key, key) + expression = @definition.expression(field) + return expression unless @definition.fields[field] == :json + + "CAST(#{ expression } AS CHAR) COLLATE utf8mb4_bin" + end + + def event_order_sql(expression) + "CASE #{ expression } WHEN 'create' THEN 0 WHEN 'update' THEN 1 " + + "WHEN 'discard' THEN 2 WHEN 'restore' THEN 3 ELSE 4 END" + end + + def sort_sql + @sort.flat_map do |key, direction| + expression = yield key + ["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"] + end + end + + def condition_labels + entries = QueryState.records(@params, 'conditions').select { _1['operator'] == 'axis_eq' } + entries += QueryState.records(@params, 'drill') + entries.group_by { _1['field'] }.to_h do |field, conditions| + values = conditions.map { _1['value'] }.compact.uniq + labels = case field + when 'tags' then tag_labels(values) + when 'parent_tag_ids' then parent_tag_labels(values) + when 'user' + actors = HistoryAccess.visible(population.scope, 'v').except(:select, :order) + .select('v.created_by_user_id') + User.where(id: values).where(id: actors) + .pluck(:id, :name).to_h.transform_keys(&:to_s) + else {} + end + [field, labels] + end + end + + def axis_labels(records, column_values) + result = {} + @axis_names = {} + axes.all.each_with_index do |axis, index| + values = records.map { _1["a#{ index }"] } + if index >= axes.rows.length + values += column_values.map { _1[index - axes.rows.length] } + end + values = values.compact.uniq - [CountsAxes::MASKED_VALUE] + labels = case axis['field'] + when 'user' + User.where(id: values).pluck(:id, :name).to_h.transform_keys(&:to_s) + when 'language_code', 'source_language_code' + values.to_h { [_1, _1] } + when 'tags' + labels = tag_labels(values) + @axis_names["a#{ index }"] = @tag_names + labels + when 'parent_tag_ids' then parent_tag_labels(values) + else {} + end + result["a#{ index }"] = labels + end + result + end + + def parent_tag_labels(values) + ids = values.flat_map { _1.to_s.split }.uniq + tag_labels(ids.map { "internal:#{ _1 }" }) + values.to_h do |value| + names = value.to_s.split.map do |id| + @tag_names.fetch("internal:#{ id }", {}).dig(locale.language_code, 'name') + end + [value.to_s, names.compact.join(' ')] + end + end + + def tag_labels(values) + internal = values.grep(/\Ainternal:/).map { _1.split(':').last.to_i } + external = values.grep(/\Aexternal:/).map { _1.split(':').last.to_i } + point = population.history ? population.at : + population.timestamp(@params['at']) || population.at + labels = {} + @tag_names = {} + sources = [[TagVersion, internal, 'internal'], [NicoTagVersion, external, 'external']] + sources.each do |model, ids, kind| + next if ids.empty? + + table = model.table_name + scope = HistoryAccess.visible(model.where(tag_id: ids)).where('created_at <= ?', point) + bound = @connection.quote(point) + if population.history && @definition.resource == 'posts' + key = kind == 'internal' ? 'tag_id' : 'external_tag_id' + pairs = HistoryAccess.visible(population.scope, 'v').except(:select, :order).joins( + "JOIN JSON_TABLE(v.tags_json, '$[*]' COLUMNS (tag_id BIGINT PATH '$.#{ key }', " + + "version_no INT PATH '$.version_no' NULL ON EMPTY)) recorded_tag") + .where('recorded_tag.tag_id IN (?)', ids).group('recorded_tag.tag_id') + value = kind == 'internal' ? 'MAX(recorded_tag.version_no)' : 'MAX(v.created_at)' + pairs = pairs.select(Arel.sql("recorded_tag.tag_id, #{ value } AS reference_point")) + scope = scope.joins("JOIN (#{ pairs.to_sql }) recorded " + + "ON recorded.tag_id = #{ table }.tag_id") + if kind == 'internal' + scope = scope.where("#{ table }.version_no = recorded.reference_point") + else + scope = scope.where("#{ table }.created_at <= recorded.reference_point") + bound = 'recorded.reference_point' + end + end + unless population.history && kind == 'internal' && @definition.resource == 'posts' + scope = scope.where("NOT EXISTS (SELECT 1 FROM #{ table } newer " + + "WHERE newer.tag_id = #{ table }.tag_id AND newer.created_at <= #{ bound } " + + "AND newer.version_no > #{ table }.version_no)") + end + scope.each do |row| + id = "#{ kind }:#{ row.tag_id }" + labels[id] = row.name + @tag_names[id] = row.respond_to?(:names) ? row.names : {} + end + end + labels + end +end diff --git a/backend/app/services/post_tag_search.rb b/backend/app/services/post_tag_search.rb new file mode 100644 index 0000000..461400d --- /dev/null +++ b/backend/app/services/post_tag_search.rb @@ -0,0 +1,30 @@ +# Shared literal parsing and reference resolution for current and snapshot post search. +class PostTagSearch + def self.literals(locale, raw) + tokens = raw.split.reject { _1.downcase == 'type:or' } + names = tokens.map { _1.sub(/\Anot:/i, '') } + registered = TagName.where(language_code: locale.language_code, name: names) + .pluck(:name, :tag_id).to_h + primary = TagName.where(language_code: locale.language_code, + tag_id: registered.values, primary_flg: true).pluck(:tag_id, :name).to_h + tokens.zip(names).map do |token, name| + { name: primary[registered[name]] || name, + negative: token.downcase.start_with?('not:') } + end + end + + def self.references(locale, literal) + references_for(locale, [literal]).fetch(literal[:name]) + end + + def self.references_for(locale, literals) + names = literals.map { _1[:name] }.uniq + internal = TagName.where(language_code: locale.language_code, name: names) + .pluck(:name, :tag_id).group_by(&:first) + external = names.empty? ? {} : ExternalTag.where( + "CONCAT(platform, ':', name) IN (?)", names) + .pluck(Arel.sql("CONCAT(platform, ':', name)"), :id).group_by(&:first) + names.to_h { |name| [name, [internal.fetch(name, []).map(&:last).uniq, + external.fetch(name, []).map(&:last).uniq]] } + end +end diff --git a/backend/app/services/query_state.rb b/backend/app/services/query_state.rb new file mode 100644 index 0000000..ce9b3de --- /dev/null +++ b/backend/app/services/query_state.rb @@ -0,0 +1,89 @@ +# Public query grammar shared with frontend/src/lib/queryState.ts. +class QueryState + SCHEMAS = { + 'conditions' => ['field', 'operator', 'value', 'end', 'blank', 'input', 'values', 'grain', 'value_type'], + 'condition_values' => ['condition', 'value', 'value_type'], + 'drill' => ['field', 'grain', 'value', 'value_type'], + 'column_order' => ['value', 'value_type'], + 'measures' => ['key', 'axis', 'value', 'value_type'] }.freeze + + def self.list(value) + return [] if value.nil? || value == '' + raise ArgumentError unless value.is_a?(String) && value.bytesize <= 32_768 + + values = value.split(/ /, -1) + raise ArgumentError if values.length > 200 || values.any?(&:empty?) || values.uniq != values + + values + end + + def self.valid_key?(key) + root, index, field, extra = key.split('.', -1) + !extra && SCHEMAS.key?(root) && index&.match?(/\A(?:0|[1-9][0-9]?)\z/) && + SCHEMAS[root].include?(field) + end + + def self.validate!(params, allowed) + raise ArgumentError if params.length > 1000 + params.each do |key, value| + raise ArgumentError unless allowed.include?(key) || valid_key?(key) + raise ArgumentError unless value.is_a?(String) && value.bytesize <= 131_072 + end + SCHEMAS.each_key { records(params, _1) } + end + + def self.records(params, key) + records = {} + params.each do |name, value| + next unless name.start_with?("#{ key }.") + raise ArgumentError unless params.key?(key) && valid_key?(name) + + _root, index, field = name.split('.') + record = records[index.to_i] ||= {} + if ['blank', 'values'].include?(field) + raise ArgumentError unless ['0', '1'].include?(value) + + value = value == '1' + elsif ['axis', 'condition'].include?(field) + raise ArgumentError unless value.match?(/\A[0-9]{1,18}\z/) + + value = value.to_i + end + record[field] = value + end + raise ArgumentError if params.key?(key) && params[key] != '' + raise ArgumentError unless records.keys.sort == (0...records.length).to_a + + records.sort.map do |_index, record| + type = record.delete('value_type') + if (['drill', 'measures', 'column_order', 'condition_values'].include?(key) || + (key == 'conditions' && record['operator'] == 'axis_eq')) + raise ArgumentError unless ['null', 'string', 'number', 'boolean'].include?(type) + + raise ArgumentError unless type == 'null' || record.key?('value') + + value = record['value'] + record['value'] = case type + when 'null' then nil + when 'number' + raise ArgumentError unless value&.match?(/\A-?[0-9]{1,18}\z/) + Integer(value, 10) + when 'boolean' + raise ArgumentError unless ['true', 'false'].include?(value) + value == 'true' + else value + end + end + record + end + end + + def self.axes(value) + list(value).map do |token| + field, grain, extra = token.split(':', -1) + raise ArgumentError if extra || !field.match?(/\A[a-z_]+\z/) || grain == '' + + grain ? { 'field' => field, 'grain' => grain } : { 'field' => field } + end + end +end diff --git a/backend/config/initializers/history_storage_access.rb b/backend/config/initializers/history_storage_access.rb new file mode 100644 index 0000000..f2d1002 --- /dev/null +++ b/backend/config/initializers/history_storage_access.rb @@ -0,0 +1,7 @@ +Rails.application.config.to_prepare do + [ActiveStorage::Blobs::ProxyController, ActiveStorage::Blobs::RedirectController, + ActiveStorage::Representations::ProxyController, + ActiveStorage::Representations::RedirectController, ActiveStorage::DiskController].each do |controller| + controller.include(HistoryStorageAccess) unless controller < HistoryStorageAccess + end +end diff --git a/backend/config/routes.rb b/backend/config/routes.rb index 01e0895..62f7f3f 100644 --- a/backend/config/routes.rb +++ b/backend/config/routes.rb @@ -1,4 +1,19 @@ Rails.application.routes.draw do + get 'counts/definitions/:resource', to: 'counts_tables#definition' + get 'counts/definitions/:resource/candidates', to: 'counts_tables#candidates' + get 'counts/query', to: 'counts_tables#index' + post 'counts/export/jobs', to: 'counts_exports#create_job' + post 'counts/export/jobs/:token/download', to: 'counts_exports#start_download' + get 'counts/export/jobs/:token', to: 'counts_exports#job_status' + delete 'counts/export/jobs/:token', to: 'counts_exports#cancel_job' + post 'counts/export/prepare', to: 'counts_exports#prepare' + get 'counts/export/files/:token', to: 'counts_exports#download' + get 'counts/export', to: 'counts_exports#index' + post 'counts/export', to: 'counts_exports#index' + get 'counts/state', to: 'counts_states#index' + get 'counts/posts', to: 'counts_posts#index' + get 'counts/tags', to: 'counts_tags#index' + resources :locales, only: [:index] resources :sitemap_entries, only: [:index, :show], param: :resource diff --git a/backend/lib/tasks/counts_exports.rake b/backend/lib/tasks/counts_exports.rake new file mode 100644 index 0000000..1a4e659 --- /dev/null +++ b/backend/lib/tasks/counts_exports.rake @@ -0,0 +1,15 @@ +namespace :counts do + desc 'Process the persistent Counts export queue (run under a process supervisor)' + task export_worker: :environment do + stopping = false + ['TERM', 'INT'].each { |signal| Signal.trap(signal) { stopping = true } } + at_exit do + path = CountsExportJobs.directory.join("#{ CountsExportJobs.worker_id }.json") + File.unlink(path) if path.exist? + end + until stopping + worked = Rails.application.executor.wrap { CountsExportJobs.work_one } + sleep(1) unless worked + end + end +end diff --git a/backend/spec/models/history_access_spec.rb b/backend/spec/models/history_access_spec.rb new file mode 100644 index 0000000..c087c35 --- /dev/null +++ b/backend/spec/models/history_access_spec.rb @@ -0,0 +1,130 @@ +require 'rails_helper' + +RSpec.describe HistoryAccess do + include_context 'Ainu locale' + + let!(:tag) { create(:tag) } + + it 'uses the complete tag-description identity and version order at equal timestamps' do + locale + rows = ['ja', 'ain'].flat_map do |language| + ['create', 'discard', 'restore'].each_with_index.map do |event, index| + TagDescriptionVersion.create!(tag_id: tag.id, language_code: language, + version_no: index + 1, event_type: language == 'ain' ? 'update' : event, + body: "#{ language } #{ index }", created_at: Time.utc(2024)) + end + end + described_class.set(user: nil) do + expect(described_class.visible(TagDescriptionVersion.where(id: rows.map(&:id))).pluck(:id)) + .to match_array([rows[2], *rows.last(3)].map(&:id)) + expect { described_class.require!(rows.first) }.to raise_error(HistoryAccess::Forbidden) + end + described_class.set(user: create(:user, :admin)) do + expect(described_class.visible(TagDescriptionVersion.where(id: rows.map(&:id))).count).to eq(6) + end + end + + it 'invalidates public artifacts after a new discard and binds admin artifacts to their actor' do + public_metadata = described_class.set(user: nil) { described_class.authority } + admin = create(:user, :admin) + admin_metadata = described_class.set(user: admin) { described_class.authority } + TagVersion.create!(tag:, version_no: 1, event_type: 'discard', name: 'private', + category: 'general', + names: { ja: { name: 'private', script_code: 'Jpan', + auto_generated: false, aliases: [] } }, + aliases: '', parent_tag_ids: '', created_at: Time.utc(2024)) + described_class.set(user: nil) do + expect(described_class.authorised_artifact?(public_metadata)).to be(false) + expect(described_class.authorised_artifact?(admin_metadata)).to be(false) + end + described_class.set(user: create(:user, :admin)) do + expect(described_class.authorised_artifact?(admin_metadata)).to be(false) + end + described_class.set(user: admin) do + expect(described_class.authorised_artifact?(admin_metadata)).to be(true) + admin.update!(role: :guest) + expect(described_class.authorised_artifact?(admin_metadata)).to be(false) + end + end + + it 'denies a blob attached to a discarded post including its derived attachment' do + post = create(:post) + blob = ActiveStorage::Blob.create_before_direct_upload!(filename: 'private.png', + byte_size: 1, checksum: Base64.strict_encode64('checksum'), content_type: 'image/png') + ActiveStorage::Attachment.create!(name: 'thumbnail', record: post, blob:) + PostVersion.create!(post:, version_no: 1, event_type: 'discard', url: post.url, + title: 'private', tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024)) + variant = ActiveStorage::VariantRecord.create!(blob:, variation_digest: 'spec-variation') + derived = ActiveStorage::Blob.create_before_direct_upload!(filename: 'derived.png', + byte_size: 1, checksum: Base64.strict_encode64('checksum'), content_type: 'image/png') + ActiveStorage::Attachment.create!(name: 'image', record: variant, blob: derived) + described_class.set(user: nil) do + expect(described_class.blob_visible?(blob)).to be(false) + expect(described_class.blob_visible?(derived)).to be(false) + end + described_class.set(user: create(:user, :admin)) do + expect(described_class.blob_visible?(derived)).to be(true) + end + end +end + +RSpec.describe 'Counts artifact policy and external references' do + it 'rejects files created before the masked-summary policy' do + HistoryAccess.set(user: nil) do + metadata = HistoryAccess.authority.except('artifact_policy') + expect(HistoryAccess.authorised_artifact?(metadata)).to be(false) + end + end + + it 'does not expose a private linked tag name through an external history snapshot' do + tag = create(:tag, primary_name: 'private-linked-name') + TagVersion.create!(tag:, version_no: 1, event_type: 'discard', name: 'private-linked-name', + category: 'general', aliases: '', parent_tag_ids: '', created_at: Time.utc(2024), + names: { ja: { name: 'private-linked-name', script_code: 'Jpan', + auto_generated: false, aliases: [] } }) + external = create(:external_tag, name: 'public-external') + row = NicoTagVersion.create!(external_tag: external, version_no: 1, event_type: 'create', + name: 'nico:public-external', linked_tags: 'private-linked-name', + created_at: Time.utc(2024)) + post = create(:post) + snapshot = PostVersion.create!(post:, version_no: 1, event_type: 'create', + title: 'external reference', url: post.url, tags: '', + tags_json: [{ external_tag_id: external.id }], parent_post_ids: '', + created_at: Time.utc(2024, 2)) + HistoryAccess.set(user: nil) do + expect(HistoryAccess.visible(NicoTagVersion.where(id: row.id))).to be_empty + expect(HistoryAccess.visible(PostVersion.where(id: snapshot.id))).to be_empty + end + HistoryAccess.set(user: create(:user, :admin)) do + expect(HistoryAccess.visible(NicoTagVersion.where(id: row.id))).to exist + end + public_row = NicoTagVersion.create!(external_tag: external, version_no: 2, event_type: 'update', + name: 'nico:public-external', linked_tags: 'private-linked-name-extra', + created_at: Time.utc(2024, 7)) + HistoryAccess.set(user: nil) do + expect(HistoryAccess.visible(NicoTagVersion.where(id: public_row.id))).to exist + expect(HistoryAccess.visible(PostVersion.where(id: snapshot.id))).to be_empty + end + end + + it 'propagates the recorded tag version policy to its post snapshot' do + parent = create(:tag, primary_name: 'private parent') + child = create(:tag, primary_name: 'referencing child') + [[parent, 'discard', '', 'private parent'], + [child, 'create', parent.id.to_s, 'referencing child']].each do |tag, event, parents, name| + TagVersion.create!(tag:, version_no: 1, event_type: event, name:, + category: 'general', aliases: '', parent_tag_ids: parents, created_at: Time.utc(2023), + names: { ja: { name:, script_code: 'Jpan', + auto_generated: false, aliases: [] } }) + end + post = create(:post) + row = PostVersion.create!(post:, version_no: 1, event_type: 'create', title: 'snapshot', + url: post.url, tags: '', parent_post_ids: '', + tags_json: [{ tag_id: child.id, version_no: 1, name: 'referencing child', sections: [] }], + created_at: Time.utc(2024)) + HistoryAccess.set(user: nil) do + expect(HistoryAccess.visible(PostVersion.where(id: row.id))).to be_empty + end + end + +end diff --git a/backend/spec/requests/counts_download_spec.rb b/backend/spec/requests/counts_download_spec.rb new file mode 100644 index 0000000..5734869 --- /dev/null +++ b/backend/spec/requests/counts_download_spec.rb @@ -0,0 +1,124 @@ +require 'rails_helper' +require 'tmpdir' + +RSpec.describe 'Counts native GET acknowledgement', type: :request do + include ActiveSupport::Testing::TimeHelpers + around do |example| + Dir.mktmpdir('counts-download-request') do |path| + @counts_download_dir = Pathname.new(path) + example.run + end + end + + before do + allow(CountsExportDownload).to receive(:directory).and_return(@counts_download_dir) + end + + def ready_job(format, payload: 'payload') + CountsExportJobs.pulse + post '/counts/export/jobs', params: { resource: 'posts', definition: 'history', + mode: 'details', columns: 'resource_id', locale: 'ja', format: } + expect(response).to have_http_status(:accepted) + job = response.parsed_body.fetch('job') + signature = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job') + owner = signature.fetch('owner') + exporter = instance_double(CountsExporter, resource: 'posts', format:, body: [payload]) + file = HistoryAccess.set(user: nil) { CountsExportDownload.prepare(exporter, owner) } + state = CountsExportJobs.read(signature.fetch('id')) + state.merge!('status' => 'ready', 'token' => file) + CountsExportJobs.write(state) + post "/counts/export/jobs/#{ job }/download" + expect(response).to have_http_status(:ok) + [job, file, response.parsed_body.fetch('download_attempt')] + end + + ['csv', 'json'].each do |format| + it "delivers every byte of a multi-chunk #{ format } file through HTTP" do + sign_in_as(create(:user, :member)) + rows = Array.new(5000) { |index| { id: index, name: '日本語,名称' } } + payload = if format == 'json' + rows.to_json + else + "id,name\r\n" + rows.map { CSV.generate_line(_1.values) }.join + end + job, file, attempt = ready_job(format, payload:) + owner = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job').fetch('owner') + metadata = HistoryAccess.set(user: nil) { CountsExportDownload.find(file, owner) } + expected = File.binread(metadata.fetch('path')) + expect(expected.bytesize).to be > 64 * 1024 + get "/counts/export/files/#{ file }", params: { job:, attempt: } + expect(response).to have_http_status(:ok) + expect(response.body.bytesize).to eq(expected.bytesize) + expect(response.body.b).to eq(expected) + expect(response.body.b).to eq(payload.b) + get "/counts/export/jobs/#{ job }" + expect(response.parsed_body['download_status']).to eq('started') + end + + it "acknowledges only an authorised #{ format } GET body, not HEAD or acceptance" do + job, file, attempt = ready_job(format) + get "/counts/export/jobs/#{ job }" + expect(response.parsed_body['download_status']).to eq('pending') + head "/counts/export/files/#{ file }", params: { job:, attempt: } + expect(response).to have_http_status(:method_not_allowed) + get "/counts/export/jobs/#{ job }" + expect(response.parsed_body['download_status']).to eq('pending') + get "/counts/export/files/#{ file }", params: { job:, attempt: } + expect(response).to have_http_status(:ok) + expect(response.headers['Content-Disposition']).to include("counts-posts.#{ format }") + expect(response.body).to eq('payload') + get "/counts/export/jobs/#{ job }" + expect(response.parsed_body['download_status']).to eq('started') + get "/counts/export/files/#{ file }", params: { job:, attempt: } + expect(response).to have_http_status(:not_found) + end + end + + it 'rejects wrong attempts, cancellation and a newly discarded resource' do + job, file, attempt = ready_job('json') + get "/counts/export/files/#{ file }", params: { job:, attempt: 'wrong' } + expect(response).to have_http_status(:not_found) + delete "/counts/export/jobs/#{ job }" + expect(response).to have_http_status(:ok) + get "/counts/export/files/#{ file }", params: { job:, attempt: } + expect(response).to have_http_status(:not_found) + job, file, attempt = ready_job('json') + record = create(:post) + PostVersion.create!(post: record, version_no: 1, event_type: 'discard', title: 'private', + url: record.url, tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024)) + get "/counts/export/files/#{ file }", params: { job:, attempt: } + expect(response).to have_http_status(:not_found) + expect(response.body).not_to include('payload', 'private') + end + it 'does not let a second browser session inspect the job or download its file' do + job, file, attempt = ready_job('json') + other = ActionDispatch::Integration::Session.new(Rails.application) + other.get("/counts/export/jobs/#{ job }") + expect(other.response).to have_http_status(:not_found) + other.get("/counts/export/files/#{ file }", params: { job:, attempt: }) + expect(other.response).to have_http_status(:not_found) + expect(other.response.body).not_to include('payload') + end + + it 'does not acknowledge an expired attempt' do + job, file, attempt = ready_job('json') + travel_to(Time.current + 46.seconds) do + get "/counts/export/files/#{ file }", params: { job:, attempt: } + expect(response).to have_http_status(:not_found) + get "/counts/export/jobs/#{ job }" + expect(response.parsed_body['download_status']).to eq('timeout') + end + end + + it 'records a missing file without accepting the GET as started' do + job, file, attempt = ready_job('json') + owner = CountsExportJobs.verifier.verified(job, purpose: 'counts_export_job').fetch('owner') + metadata = HistoryAccess.set(user: nil) { CountsExportDownload.find(file, owner) } + File.unlink(metadata.fetch('path')) + get "/counts/export/files/#{ file }", params: { job:, attempt: } + expect(response).to have_http_status(:not_found) + get "/counts/export/jobs/#{ job }" + expect(response.parsed_body['download_status']).to eq('unavailable') + end + +end diff --git a/backend/spec/requests/counts_history_boundaries_spec.rb b/backend/spec/requests/counts_history_boundaries_spec.rb new file mode 100644 index 0000000..0bdb514 --- /dev/null +++ b/backend/spec/requests/counts_history_boundaries_spec.rb @@ -0,0 +1,40 @@ +require 'rails_helper' + +RSpec.describe 'History boundaries shared with Counts', type: :request do + let!(:tag) { create(:tag, primary_name: 'description tag') } + let!(:description) do + TagDescription.create!(tag:, language_code: 'ja', version_no: 2, body: 'private body') + end + + before do + ['create', 'discard'].each_with_index do |event, index| + TagDescriptionVersion.create!(tag_id: tag.id, language_code: 'ja', + version_no: index + 1, event_type: event, body: 'private body', + created_at: Time.utc(2024)) + end + end + + it 'denies the same discarded description through Counts, the facade and the ordinary API' do + get '/counts/query', params: { resource: 'tag_descriptions', definition: 'history', + mode: 'details', locale: 'ja', execution_at: '2025-01-01T00:00:00Z' } + expect(response).to have_http_status(:ok) + expect(response.parsed_body).to include('count' => 0, 'output_count' => 2, + 'excluded_count' => 2, 'versions' => []) + ["/wiki/#{ description.id }", "/tags/#{ tag.id }/description"].each do |path| + get path, params: { locale: 'ja' } + expect(response).to have_http_status(:not_found) + expect(response.body).not_to include('private body') + end + get '/sitemap_entries/descriptions', params: { locale: 'ja' } + expect(response).to have_http_status(:ok) + expect(response.parsed_body['ids']).not_to include(description.id) + end + + it 'permits the server-authenticated administrator to read the discarded description' do + sign_in_as(create(:user, :admin)) + get "/tags/#{ tag.id }/description", params: { locale: 'ja' } + expect(response).to have_http_status(:ok) + expect(response.parsed_body['body']).to eq('private body') + expect(response.headers['Cache-Control']).to include('no-store') + end +end diff --git a/backend/spec/requests/counts_spec.rb b/backend/spec/requests/counts_spec.rb new file mode 100644 index 0000000..0026d49 --- /dev/null +++ b/backend/spec/requests/counts_spec.rb @@ -0,0 +1,97 @@ +require 'rails_helper' + +RSpec.describe 'Legacy Counts API', type: :request do + let!(:actor) { create(:user, :member) } + let!(:post) { create(:post) } + let!(:tag) { create(:tag, primary_name: 'counts tag', category: :general) } + + before do + snapshot = { tag_id: tag.id, version_no: 1, name: 'counts tag', + category: 'general', sections: [] } + [1, 2, 3].each do |number| + time = Time.utc(2024, number == 3 ? 2 : 1, number, 0, 0) + PostVersion.create!(post:, version_no: number, + event_type: number == 1 ? 'create' : 'update', + tags: '', url: post.url, title: "Snapshot #{ number }", parent_post_ids: '', + tags_json: number == 1 ? [snapshot] : [], + created_by_user: number == 3 ? nil : actor, created_at: time) + TagVersion.create!(tag:, version_no: number, + event_type: number == 1 ? 'create' : 'update', + name: 'counts tag', category: 'general', aliases: '', parent_tag_ids: '', + names: { ja: { name: 'counts tag', script_code: 'Jpan', + auto_generated: false, aliases: [] } }, + created_by_user: number == 3 ? nil : actor, created_at: time) + end + end + + ['posts', 'tags'].each do |resource| + context resource do + let(:path) { "/counts/#{ resource }" } + let(:id) { resource == 'posts' ? post.id : tag.id } + let(:filters) { { resource_id: id, user_id: actor.id, latest: '0', + from: '2024-01-01T00:00Z', before: '2024-02-01T00:00Z' } } + + it 'uses the same AND filters for SQL totals, groups and details' do + get path, params: filters.merge(axis: 'month') + expect(response).to have_http_status(:ok) + summary = response.parsed_body + expect(summary['count']).to eq(2) + expect(summary['groups'].sum { _1['count'] }).to eq(2) + expect(summary['groups'].map { _1['key'] }).to eq(['2024-01']) + + get path, params: filters.merge(mode: 'details') + expect(response.parsed_body['count']).to eq(2) + expect(response.parsed_body['versions'].map { _1['version_no'] }).to eq([2, 1]) + end + + it 'paginates details and groups independently from totals' do + get path, params: { resource_id: id, latest: '0', mode: 'details', limit: 1, page: 2 } + expect(response.parsed_body['count']).to eq(3) + expect(response.parsed_body['versions'].sole['version_no']).to eq(2) + get path, params: { resource_id: id, latest: '0', axis: 'month', limit: 1, page: 2 } + expect(response.parsed_body['group_count']).to eq(2) + expect(response.parsed_body['groups'].sole).to include('key' => '2024-02', 'count' => 1) + end + + it 'groups null actors and supports their drill-down filter' do + get path, params: { resource_id: id, latest: '0', axis: 'user' } + expect(response.parsed_body['groups']).to include( + { 'key' => nil, 'name' => nil, 'count' => 1 }) + get path, params: { resource_id: id, mode: 'details', user_id: 'none' } + expect(response.parsed_body['count']).to eq(1) + end + + it 'uses an inclusive start and exclusive end' do + get path, params: { resource_id: id, latest: '0', + from: '2024-01-01T00:00Z', before: '2024-01-02T00:00Z' } + expect(response.parsed_body['count']).to eq(1) + end + + it 'rejects unsupported or malformed query values' do + [ + { axis: 'SQL' }, { limit: 101 }, { page: '1x' }, { user_id: '-1' }, + { event_type: 'invalid' }, { columns: 'password' }, { order: 'id' }, + { from: '2024-02-30T00:00Z' }, { from: '2024-01-01' }, + { user_id: ['1'] }, { mode: 'invalid' }].each do |invalid| + get path, params: invalid + expect(response).to have_http_status(:bad_request) + end + end + + it 'retains the existing banned-user boundary' do + actor.update!(banned_at: Time.current) + sign_in_as(actor) + get path + expect(response).to have_http_status(:forbidden) + end + end + end + + it 'matches removed tags from the previous snapshot, not current post_tags' do + get '/counts/posts', params: { tag: tag.id, latest: '0', mode: 'details' } + expect(response.parsed_body['count']).to eq(2) + expect(response.parsed_body['versions'].map { _1['version_no'] }).to eq([2, 1]) + expect(response.parsed_body['versions'].first['tags']).to be_empty + expect(response.parsed_body['versions'].last['tags'].sole['id']).to eq(tag.id) + end +end diff --git a/backend/spec/requests/counts_table_spec.rb b/backend/spec/requests/counts_table_spec.rb new file mode 100644 index 0000000..5e13aac --- /dev/null +++ b/backend/spec/requests/counts_table_spec.rb @@ -0,0 +1,376 @@ +require 'rails_helper' + +RSpec.describe 'Counts table API', type: :request do + include_context 'Ainu locale' + + let!(:actor) { create(:user, :member, name: 'Counts actor') } + let!(:record) { create(:post) } + let!(:tag) { create(:tag, primary_name: 'alpha', category: :general) } + let!(:external) { create(:external_tag, name: 'beta') } + let(:point) { '2025-01-01T00:00:00Z' } + let(:base) do + { resource: 'posts', definition: 'history', mode: 'details', + execution_at: point, timezone: 'UTC', locale: 'ja' } + end + + before do + TagVersion.create!(tag:, version_no: 1, event_type: 'create', name: 'alpha', + category: 'general', aliases: '', parent_tag_ids: '', + names: { ja: { name: 'alpha', script_code: 'Jpan', auto_generated: false, aliases: [] } }, + created_at: Time.utc(2023)) + NicoTagVersion.create!(external_tag: external, version_no: 1, + event_type: 'create', name: 'beta', linked_tags: '', created_at: Time.utc(2023)) + snapshots = [ + { tag_id: tag.id, version_no: 1, name: 'alpha', category: 'general', sections: [] }, + { external_tag_id: external.id }] + [1, 2, 3].each do |number| + PostVersion.create!(post: record, version_no: number, + event_type: number == 1 ? 'create' : 'update', url: record.url, + title: "snapshot #{ number }", tags: '', parent_post_ids: '', + tags_json: snapshots.first(number == 1 ? 2 : (number == 2 ? 1 : 0)), + created_by_user: number == 3 ? nil : actor, + created_at: Time.utc(2024, 1, 31, 15) + (number - 1).hours) + end + end + + def query(extra = {}) + get '/counts/query', params: base.merge(extra) + expect(response).to have_http_status(:ok) + response.parsed_body + end + + def axis(field, value, type: 'string', grain: nil, root: 'conditions') + result = { root => '', "#{ root }.0.field" => field, + "#{ root }.0.value_type" => type } + result["#{ root }.0.operator"] = 'axis_eq' if root == 'conditions' + result["#{ root }.0.value"] = value unless type == 'null' + result["#{ root }.0.grain"] = grain if grain + result + end + + def versions(result) = result.fetch('versions').map { _1.fetch('version_no') } + + it 'identifies a totals timeout without exposing SQL in the error response' do + allow_any_instance_of(CountsPivotTotals).to receive(:call) + .and_raise(CountsPivotTotals::TimedOut, 'private SQL') + get '/counts/query', params: base.merge(mode: 'summary', rows: 'created_at:month') + expect(response).to have_http_status(:service_unavailable) + expect(response.parsed_body).to eq('error' => 'counts_query_timeout', + 'stage' => 'totals', 'timeout_seconds' => 30) + expect(response.body).not_to include('private SQL') + end + + it 'applies the same optimizer safeguards to the paginated detail SELECT' do + statements = [] + listener = lambda do |_name, _start, _finish, _id, payload| + statements << payload[:sql] + end + ActiveSupport::Notifications.subscribed(listener, 'sql.active_record') do + result = query(tags: 'alpha', limit: 1) + expect(result['count']).to eq(2) + expect(versions(result)).to eq([1]) + end + details = statements.find { _1.include?('AS counts_latest') } + expect(details).not_to be_nil + expect(details).to include('MAX_EXECUTION_TIME(30000)', + "SET_VAR(optimizer_switch='semijoin=off')") + end + + it 'separates latest and history and defaults to stable detail ordering and limit 100' do + history = query + expect(versions(history)).to eq([1, 2, 3]) + expect(history).to include('limit' => 100, 'count' => 3, 'output_count' => 3, + 'visible_output_count' => 3, 'excluded_count' => 0) + expect(history['versions'].map { _1['latest'] }).to eq([false, false, true]) + expect(versions(query(definition: 'latest'))).to eq([3]) + expect(versions(query(limit: 1, page: 2))).to eq([2]) + expect(versions(query(sort: 'version_no:desc', limit: 1))).to eq([3]) + end + + it 'counts regular pivot rows separately from totals and from underlying records' do + total = query(mode: 'summary', rows: '', cols: '') + expect(total).to include('count' => 3, 'output_count' => 1, 'group_count' => 1) + expect(total['cells']).to eq([{ 'values' => [], 'count' => 3 }]) + pivot = query(mode: 'summary', rows: 'event_type', cols: '', sort: '', limit: 1) + expect(pivot).to include('group_count' => 2, 'visible_output_count' => 2) + expect(pivot['row_headers'].count { _1['kind'] == 'data' }).to eq(1) + expect(pivot['row_headers']).to include(hash_including('kind' => 'grand_total')) + expect(query(mode: 'summary', rows: '', cols: 'event_type', sort: '')['output_count']).to eq(1) + end + + it 'preserves timezone, NULL and legacy drill semantics and permits condition removal' do + month = axis('created_at', '2024-02', grain: 'month') + expect(versions(query(month.merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3]) + expect(versions(query(month))).to eq([]) + old = axis('created_at', '2024-02', grain: 'month', root: 'drill') + expect(versions(query(old.merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3]) + expect(versions(query(old.merge(month).merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3]) + expect(versions(query(axis('user', nil, type: 'null')))).to eq([3]) + expect(versions(query(axis('changed_field', 'title')))).to eq([2, 3]) + expect(versions(query)).to eq([1, 2, 3]) + end + + it 'evaluates latest before filtering the version set' do + predicate = { conditions: '', 'conditions.0.field' => 'latest', + 'conditions.0.operator' => 'eq', 'conditions.0.value' => 'false' } + expect(versions(query(predicate))).to eq([1, 2]) + expect(versions(query(predicate.merge('conditions.0.value' => 'true')))).to eq([3]) + end + + it 'distinguishes the snapshot cutoff from the current latest flag and condition' do + past = { definition: 'latest', at: '2024-01-31T16:00:00Z' } + result = query(past) + expect(versions(result)).to eq([2]) + expect(result['versions'].first['latest']).to be(false) + predicate = { conditions: '', 'conditions.0.field' => 'latest', + 'conditions.0.operator' => 'eq', 'conditions.0.value' => 'false' } + expect(versions(query(past.merge(predicate)))).to eq([2]) + expect(versions(query(past.merge(predicate).merge('conditions.0.value' => 'true')))) + .to eq([]) + unchanged = query(definition: 'latest', at: '2024-02-01T00:00:00Z') + expect(versions(unchanged)).to eq([3]) + expect(unchanged['versions'].first['latest']).to be(true) + expect(versions(query(sort: 'latest:desc resource_id:asc version_no:asc'))).to eq([3, 1, 2]) + end + + it 'checks the current latest flag beyond the execution timestamp' do + PostVersion.create!(post: record, version_no: 4, event_type: 'update', + url: record.url, title: 'after execution', tags: '', tags_json: [], + parent_post_ids: '', created_at: Time.utc(2025, 2, 1)) + result = query + expect(versions(result)).to eq([1, 2, 3]) + expect(result['versions'].map { _1['latest'] }).to eq([false, false, false]) + snapshot = query(definition: 'latest') + expect(versions(snapshot)).to eq([3]) + expect(snapshot['versions'].first['latest']).to be(false) + end + + it 'keeps ANY, ALL and NONE distinct without multiplying parent records' do + values = { conditions: '', 'conditions.0.field' => 'tags', + 'conditions.0.values' => '1', condition_values: '', + 'condition_values.0.condition' => '0', + 'condition_values.0.value_type' => 'string', + 'condition_values.0.value' => "internal:#{ tag.id }", + 'condition_values.1.condition' => '0', + 'condition_values.1.value_type' => 'string', + 'condition_values.1.value' => "external:#{ external.id }" } + { 'any' => [1, 2], 'all' => [1], 'none' => [3] }.each do |operator, expected| + expect(versions(query(values.merge('conditions.0.operator' => operator)))).to eq(expected) + end + result = query(axis('tags', "internal:#{ tag.id }")) + expect(result['versions'].first['tags'].length).to eq(2) + end + + it 'distinguishes IN, NOT IN, an empty selection and an unspecified selection' do + predicate = { conditions: '', 'conditions.0.field' => 'version_no', + 'conditions.0.operator' => 'in', 'conditions.0.values' => '1', + condition_values: '', 'condition_values.0.condition' => '0', + 'condition_values.0.value_type' => 'number', + 'condition_values.0.value' => '1' } + expect(versions(query(predicate))).to eq([1]) + expect(versions(query(predicate.merge('conditions.0.operator' => 'not_in')))).to eq([2, 3]) + empty = predicate.reject { |key, _value| key.to_s.start_with?('condition_values.') } + expect(versions(query(empty))).to be_empty + expect(versions(query(empty.except('conditions.0.values')))).to eq([1, 2, 3]) + end + + it 'resolves current tag names for AND, OR and negative searches' do + expect(versions(query(tags: 'alpha'))).to eq([1, 2]) + expect(versions(query(tags: 'alpha nico:beta'))).to eq([1]) + expect(versions(query(tags: 'nico:beta alpha'))).to eq([1]) + expect(versions(query(tags: 'alpha not:nico:beta'))).to eq([2]) + expect(versions(query(tags: 'alpha nico:beta', match: 'any'))).to eq([1, 2]) + expect(versions(query(tags: 'not:alpha'))).to eq([3]) + end + + context 'when current tag names differ from recorded snapshot names' do + before do + tag.tag_names.find_by!(language_code: 'ja', primary_flg: true) + .update!(name: 'renamed-alpha') + create(:tag_name, :alias, tag:, name: 'alpha-alias') + TagVersion.create!(tag:, version_no: 2, event_type: 'update', name: 'renamed-alpha', + category: 'general', aliases: 'alpha-alias', parent_tag_ids: '', + names: { ja: { name: 'renamed-alpha', script_code: 'Jpan', auto_generated: false, + aliases: [{ name: 'alpha-alias', script_code: 'Jpan' }] } }, + created_at: Time.utc(2024, 2, 1)) + PostVersion.create!(post: record, version_no: 4, event_type: 'update', + url: record.url, title: 'after rename', tags: '', parent_post_ids: '', + tags_json: [{ tag_id: tag.id, version_no: 2, name: 'renamed-alpha', + category: 'general', sections: [] }], + created_by_user: actor, created_at: Time.utc(2024, 2, 2)) + end + + it 'resolves current names and aliases to stable IDs in both definitions' do + { 'history' => [1, 2, 4], 'latest' => [4] }.each do |definition, expected| + ['renamed-alpha', 'alpha-alias'].each do |name| + expect(versions(query(tags: name, definition:))).to eq(expected) + end + end + result = query(tags: 'renamed-alpha') + expect(result['versions'].first['tags'].first['name']).to eq('alpha') + expect(versions(query(tags: 'renamed-alpha', definition: 'latest', + at: '2024-01-31T16:00:00Z'))).to eq([2]) + expect(versions(query(tags: 'not:alpha-alias'))).to eq([3]) + expect(versions(query(tags: 'renamed-alpha nico:beta'))).to eq([1]) + expect(versions(query(tags: 'renamed-alpha nico:beta', match: 'any'))).to eq([1, 2, 4]) + end + + it 'does not match a historical name reassigned to another tag ID' do + create(:tag, primary_name: 'alpha') + expect(versions(query(tags: 'alpha'))).to eq([]) + expect(versions(query(tags: 'unregistered-name'))).to eq([]) + expect(versions(query(tags: 'not:unregistered-name'))).to eq([1, 2, 3, 4]) + expect(versions(query(tags: 'renamed-alpha'))).to eq([1, 2, 4]) + end + + it 'resolves external names from current registrations too' do + external.update!(name: 'renamed-beta') + expect(versions(query(tags: 'nico:renamed-beta'))).to eq([1]) + expect(versions(query(tags: 'nico:beta'))).to eq([]) + expect(versions(query(tags: 'not:nico:renamed-beta'))).to eq([2, 3, 4]) + end + end + + it 'rejects malformed structured conditions and axes without accepting arbitrary SQL' do + [ + { rows: '[{"field":"user"}]' }, { rows: 'user user' }, + { rows: 'user:month' }, { rows: 'password' }, { limit: 101 }, { limit: 0 }, + { page: '1x' }, { conditions: 'eyJmaWVsZCI6InVzZXIifQ==' }, + { conditions: '', 'conditions.1.field' => 'user' }, + axis('password', '1'), axis('changed_field', 'title').merge(definition: 'latest'), + axis('user', 'yes', type: 'boolean'), + { conditions: '', 'conditions.0.field' => 'title', + 'conditions.0.operator' => 'eq', 'conditions.0.value' => 'x' * 8193 }].each do |invalid| + get '/counts/query', params: base.merge(invalid) + expect(response).to have_http_status(:bad_request), invalid.inspect + end + end + + it 'rejects duplicate URL keys and JSON query payloads at the HTTP boundary' do + ['resource=posts&tag_refs=', 'resource=posts&tag_refs.0.tag_id=1', + 'resource=posts&limit=20&limit=50', 'resource=posts&headers=%5B%22Id.%22%5D', + 'resource=posts&limit%5B%5D=20'].each do |query_string| + get "/counts/query?#{ query_string }" + expect(response).to have_http_status(:bad_request) + end + end + + it 'reports column overflow separately from malformed input' do + 101.times do |index| + PostVersion.create!(post: record, version_no: index + 4, + event_type: 'update', url: record.url, title: "column #{ index }", + tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 3)) + end + get '/counts/query', params: base.merge(mode: 'summary', rows: '', cols: 'title', sort: '') + expect(response).to have_http_status(:unprocessable_entity) + expect(response.parsed_body).to eq('error' => 'too_many_columns', 'max_columns' => 100) + end + + it 'defines all six resources with filterable output fields and shared detail defaults' do + locale + CountsResourceDefinition::DEFINITIONS.each_key do |resource| + get "/counts/definitions/#{ resource }", params: { locale: 'ain-Kana' } + expect(response).to have_http_status(:ok) + body = response.parsed_body + filters = body['filter_fields'].map { _1['key'] } + walk = lambda do |fields, prefix| + fields.each do |field| + key = [prefix, field['key']].compact.join('.') + expect(filters).to include(key) + walk.call(field.fetch('children', []), key) + end + end + walk.call(body['output_fields'], nil) + defaults = body['defaults'].select { _1['mode'] == 'details' } + expect(defaults.map { _1['definition'] }).to eq(['latest', 'history']) + expect(defaults.first['params']).to eq(defaults.last['params']) + settings = defaults.first['params'].to_h { [_1['key'], _1['value']] } + expect(settings['sort']).to eq('resource_id:asc version_no:asc') + if resource == 'posts' + expect(settings['children'].split).to eq(['tags.name_ain', 'parents.id']) + end + keys = body['name_columns'].map { _1['key'] } + languages = Language.order(:created_at, :code).pluck(:code) + expect(keys).to eq(languages.map { "name_#{ _1 }" } + languages.map { "aliases_#{ _1 }" }) + end + end + + [nil, :guest, :member, :admin].each do |role| + context "authority #{ role || 'anonymous' }" do + before do + sign_in_as(create(:user, role)) if role + # Equal timestamps deliberately require version ordering, not a date comparison. + ['discard', 'restore', 'discard', 'restore'].each_with_index do |event, index| + PostVersion.create!(post: record, version_no: index + 4, event_type: event, + url: record.url, title: event == 'restore' ? 'restored' : 'private snapshot', + tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 3)) + end + end + + it 'keeps every earlier discard boundary private but counts denied records' do + result = query + expected = role == :admin ? (1..7).to_a : [7] + expect(versions(result)).to eq(expected) + expect(result).to include('output_count' => 7, + 'visible_output_count' => expected.length, 'excluded_count' => 7 - expected.length) + expect(response.headers['Cache-Control']).to include('no-store') + expect(response.body).not_to include('private snapshot') unless role == :admin + end + + it 'applies the same version boundary to the ordinary history endpoint and old values' do + get '/posts/versions', params: { post: record.id, locale: 'ja' } + expect(response).to have_http_status(:ok) + rows = response.parsed_body.fetch('versions') + expect(rows.map { _1['version_no'] }).to eq(role == :admin ? (1..7).to_a.reverse : [7]) + unless role == :admin + expect(rows.sole.dig('title', 'prev')).to be_nil + expect(response.body).not_to include('private snapshot') + end + end + + it 'does not resolve private condition names even when the result set is empty' do + TagVersion.create!(tag:, version_no: 2, event_type: 'discard', name: 'secret tag', + category: 'general', aliases: '', parent_tag_ids: '', + names: { ja: { name: 'secret tag', script_code: 'Jpan', auto_generated: false, aliases: [] } }, + created_at: Time.utc(2024, 2)) + result = query(axis('tags', "internal:#{ tag.id }").merge(definition: 'latest')) + expect(result['versions']).to be_empty + expected = role == :admin ? { "internal:#{ tag.id }" => 'secret tag' } : {} + expect(result['condition_labels']['tags']).to eq(expected) + get '/counts/definitions/posts/candidates', params: { + field: 'tags', query: 'secret', execution_at: point, locale: 'ja' } + expect(response).to have_http_status(:ok) + expect(response.parsed_body['rows']).to be_empty + end + end + end + it 'includes private history in the public timeline without an exclusion count' do + PostVersion.create!(post: record, version_no: 4, event_type: 'discard', url: record.url, + title: 'hidden title', tags: '', tags_json: [], parent_post_ids: '', + created_at: Time.utc(2024, 3)) + result = query(mode: 'summary', rows: 'created_at:month', cols: 'event_type') + expect(result['excluded_count']).to be_nil + expect(result['count']).to eq(4) + expect(result['cells'].sum { _1['count'] }).to eq(4) + expect(response.body).not_to include('hidden title') + masked = query(mode: 'summary', rows: 'resource_id', cols: '') + expect(masked['cells'].map { _1['values'] }).to eq([[CountsAxes::MASKED_VALUE]]) + expect(masked['cells'].first['count']).to eq(4) + expect(query['versions']).to be_empty + end + + it 'rejects private summary predicates instead of exposing a count oracle' do + get '/counts/query', params: base.merge(mode: 'summary', resource_id: record.id.to_s) + expect(response).to have_http_status(:forbidden) + expect(json).to eq('error' => 'private_condition') + end + + it 'reports the combined axis limit without replacing it with a generic 400' do + get '/counts/query', params: base.merge(mode: 'summary', sort: '', + rows: 'resource_id version_no event_type user created_at:month', + cols: 'title url thumbnail_base video_ms') + expect(response).to have_http_status(:bad_request) + expect(response.parsed_body).to eq('error' => 'too_many_axes', 'max_axes' => 8) + end + +end diff --git a/backend/spec/requests/post_locale_filter_identity_spec.rb b/backend/spec/requests/post_locale_filter_identity_spec.rb index 99a6416..1b55e1e 100644 --- a/backend/spec/requests/post_locale_filter_identity_spec.rb +++ b/backend/spec/requests/post_locale_filter_identity_spec.rb @@ -11,34 +11,49 @@ RSpec.describe 'Post filter identity across locales', type: :request do let!(:other_post) { create(:post) } before do - # Legacy data can predate the current prohibition on internal nico: names. - primary_tag_name_for(tag, 'ja').update_columns(name: 'nico:shared') primary_tag_name_for(tag, 'ain').update!(name: 'シロシ', auto_generated: false) create(:tag_name, :alias, tag:, name: 'union_alias', language_code: 'ja') PostTag.create!(post: internal_post, tag:) PostExternalTag.create!(post: external_post, external_tag: external) end - ['nico:shared', 'union_alias', 'not:nico:shared'].each do |literal| + literals = ['legacy_union_source', 'union_alias', 'not:legacy_union_source', 'nico:shared'] + literals.each do |literal| it "preserves the complete match set for #{ literal }" do get '/posts', params: { locale: 'ja', tags: literal } expect(response).to have_http_status(:ok) original_ids = json.fetch('posts').map { _1.fetch('id') } - expected = literal.start_with?('not:') ? [other_post.id] : - [internal_post.id, external_post.id] + expected = if literal.start_with?('not:') + [external_post.id, other_post.id] + elsif literal == 'nico:shared' + [external_post.id] + else + [internal_post.id] + end expect(original_ids).to match_array(expected) get '/posts/tag_filters', params: { locale: 'ja', target_locale: 'ain-Kana', tags: literal } expect(response).to have_http_status(:ok) - references = json.fetch('references') - expect(references.first).to include('tag_id' => tag.id, 'external_tag_id' => external.id) - - get '/posts', params: { - locale: 'ain-Kana', tags: references.map { _1.fetch('name') }.join(' '), - tag_refs: references.to_json } + translated = json.fetch('tags') + expect(translated).to eq(literal == 'nico:shared' ? literal : + (literal.start_with?('not:') ? 'not:シロシ' : 'シロシ')) + get '/posts', params: { locale: 'ain-Kana', tags: translated } expect(response).to have_http_status(:ok) expect(json.fetch('posts').map { _1.fetch('id') }).to match_array(original_ids) end end + + it 'rejects a locale conversion that cannot preserve a legacy union' do + primary_tag_name_for(tag, 'ja').update_columns(name: 'nico:shared') + get '/posts/tag_filters', params: { + locale: 'ja', target_locale: 'ain-Kana', tags: 'nico:shared' } + expect(response).to have_http_status(:unprocessable_entity) + end + + it 'rejects removed reference parameters instead of overriding the name' do + get '/posts', params: { locale: 'ja', tags: 'legacy_union_source', + tag_refs: '', 'tag_refs.0.tag_id' => external.id.to_s } + expect(response).to have_http_status(:bad_request) + end end diff --git a/backend/spec/requests/tag_description_wiki_spec.rb b/backend/spec/requests/tag_description_wiki_spec.rb index 5a52675..19b0e19 100644 --- a/backend/spec/requests/tag_description_wiki_spec.rb +++ b/backend/spec/requests/tag_description_wiki_spec.rb @@ -55,7 +55,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do end it 'does not fall back to a revision absent from new history' do - versions.first.destroy! + TagDescriptionVersion.where(id: versions.first.id).delete_all without_legacy_reads do get "/wiki/#{ description.id }", params: { version: versions.first.id } end @@ -63,7 +63,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do end it 'does not fall back when the new current revision is missing' do - versions.last.destroy! + TagDescriptionVersion.where(id: versions.last.id).delete_all without_legacy_reads { get "/wiki/#{ description.id }" } expect(response).to have_http_status(:not_found) end @@ -190,7 +190,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do describe 'index' do [nil, 'description_contract'].each do |title| it "returns new content once in #{ title ? 'filtered' : 'unfiltered' } index" do - get '/wiki', params: { title: } + get '/wiki', params: title.nil? ? { } : { title: title } expect(response).to have_http_status(:ok) rows = json.select { |row| row.fetch('id') == description.id } expect(rows.size).to eq(1) diff --git a/backend/spec/services/counts_export_spec.rb b/backend/spec/services/counts_export_spec.rb new file mode 100644 index 0000000..de205d1 --- /dev/null +++ b/backend/spec/services/counts_export_spec.rb @@ -0,0 +1,264 @@ +require 'rails_helper' +require 'tmpdir' +require 'csv' + +RSpec.describe 'Counts export contracts' do + include ActiveSupport::Testing::TimeHelpers + let(:payload) do + { 'resource' => 'posts', 'definition' => 'history', 'mode' => 'details', + 'locale' => 'ja', 'execution_at' => '2024-03-01T00:00:00Z', + 'columns' => 'title', 'headers' => '["Title"]', 'format' => 'json' } + end + + around do |example| + Dir.mktmpdir('counts-export-spec') do |path| + @counts_export_dir = Pathname.new(path) + HistoryAccess.set(user: nil) { example.run } + end + end + + before do + allow(CountsExportDownload).to receive(:directory) + .and_return(@counts_export_dir) + end + + context 'serialization after the authorised query' do + let(:rows) { [{ title: "=formula\n\"quoted\"" }, { title: 'second' }] } + let(:page) do + { count: 2, output_count: 2, visible_output_count: 2, page: 1, limit: 100, + versions: rows, execution_at: payload.fetch('execution_at') } + end + + before do + query = CountsTableQuery.new(payload.slice(*CountsTableQuery::KEYS)) + # SQL/authority are exercised in request specs; this unit boundary supplies their page. + allow(query).to receive(:each_page) do |before_page:, &block| + before_page.call + block.call(page) + end + allow(CountsTableQuery).to receive(:new).and_return(query) + end + + it 'reports aggregation, actual output progress and regular-row metadata' do + progress = [] + phases = [] + body = CountsTableExport.new(payload).body( + progress: ->(done, total) { progress << [done, total] }, + phase: ->(value) { phases << value }).to_a.join + result = JSON.parse(body) + expect(result['data']).to eq(rows.map(&:stringify_keys)) + expect(result['metadata']).to include('logical_count_unit' => 'regular_result_rows', + 'aggregate_records_in_logical_count' => false, 'output_count' => 2) + expect(phases).to eq(['aggregating', 'writing']) + expect(progress).to include([0, 2], [1, 2], [2, 2]) + expect(progress.last).to eq([2, 2]) + end + + it 'preserves CSV multiline text and quotes while neutralising spreadsheet formulas' do + body = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join + expect(CSV.parse(body.delete_prefix("\uFEFF"))) + .to eq([['Title'], ["'=formula\n\"quoted\""], ['second']]) + end + + it 'completes an isolated worker job with its real processed count and cancellable file' do + CountsExportJobs.pulse + token = CountsExportJobs.start(payload, 'owner', 'ja') + id = CountsExportJobs.authorised_id(token, 'owner') + state = CountsExportJobs.read(id).merge('status' => 'running', 'phase' => 'aggregating') + CountsExportJobs.write(state) + # Invoke the worker entrypoint directly; never fork or start an external worker in specs. + CountsExportJobs.perform(state) + ready = CountsExportJobs.status(token, 'owner') + expect(ready).to include('status' => 'ready', 'processed' => 2, 'total' => 2) + file = CountsExportDownload.find(ready.fetch('token'), 'owner') + expect(JSON.parse(File.read(file.fetch('path')))['data'].length).to eq(2) + CountsExportJobs.status(token, 'owner', cancel: true) + expect(CountsExportDownload.find(ready.fetch('token'), 'owner')).to be_nil + end + + it 'marks a generation exception as failed without issuing a download token' do + CountsExportJobs.pulse + token = CountsExportJobs.start(payload, 'owner', 'ja') + id = CountsExportJobs.authorised_id(token, 'owner') + state = CountsExportJobs.read(id).merge('status' => 'running') + CountsExportJobs.write(state) + allow(CountsExportDownload).to receive(:prepare).and_raise(CountsExportDownload::TooLarge) + CountsExportJobs.perform(state) + expect(CountsExportJobs.status(token, 'owner')).to include('status' => 'failed') + expect(CountsExportJobs.status(token, 'owner')).not_to have_key('token') + end + end + + it 'retains multilingual names and all inner rows in JSON with one selected display column' do + names = [{ language_code: 'ja', name: 'alpha', script_code: 'Jpan', aliases: [] }, + { language_code: 'ain', name: 'recorded', script_code: 'Kana', + aliases: [{ name: 'alias', script_code: 'Latn' }] }] + rows = [{ id: 1, name_ja: 'alpha', names: }, { id: 2, name_ja: 'beta', names: }] + projection = CountsExportProjection.new(payload.merge('columns' => 'tags', + 'headers' => '["Tags"]', 'children' => 'tags.name_ja'), + resource: 'posts', summary: false, tags: false) + result = projection.project({ tags: rows }) + expect(result['tags'].length).to eq(2) + expect(result['tags'].map { _1['name_ja'] }).to eq(['alpha', 'beta']) + expect(result['tags'].map { _1['names'] }).to eq([names, names]) + end + + it 'requires a live worker, enforces queue capacity and binds jobs to their owner' do + expect { CountsExportJobs.start(payload, 'owner', 'ja') } + .to raise_error(CountsExportJobs::WorkerUnavailable) + CountsExportJobs.pulse + token = CountsExportJobs.start(payload, 'owner', 'ja') + expect(CountsExportJobs.status(token, 'other')).to be_nil + expect(CountsExportJobs.status(token, 'owner')).to include( + 'status' => 'queued', 'phase' => 'queued', 'processed' => 0, 'total' => nil) + (CountsExportDownload::MAX_CONCURRENT_EXPORTS - 1).times do + CountsExportJobs.start(payload, 'owner', 'ja') + end + expect { CountsExportJobs.start(payload, 'owner', 'ja') } + .to raise_error(CountsExportDownload::Busy) + expect(CountsExportJobs.status(token, 'owner', cancel: true)['status']).to eq('cancelled') + end + + it 'fails stale queued/running jobs and preserves actual progress without a fake total' do + CountsExportJobs.pulse + token = CountsExportJobs.start(payload, 'owner', 'ja') + id = CountsExportJobs.authorised_id(token, 'owner') + state = CountsExportJobs.read(id) + state.merge!('status' => 'running', 'phase' => 'aggregating') + CountsExportJobs.write(state) + CountsExportJobs.checkpoint(id, 5, nil, phase: 'writing') + expect(CountsExportJobs.status(token, 'owner')).to include( + 'phase' => 'writing', 'processed' => 5, 'total' => nil) + state = CountsExportJobs.read(id) + state['heartbeat_at'] = (Time.current - CountsExportJobs::HEARTBEAT_SECONDS - 1).to_f + CountsExportJobs.write(state) + expect(CountsExportJobs.status(token, 'owner')).to include( + 'status' => 'failed', 'error' => 'worker_stalled', 'processed' => 5) + expect { CountsExportJobs.checkpoint(id, 6) }.to raise_error(CountsExportJobs::Cancelled) + end + + it 'revokes an already generated download on a new discard and rejects wrong owners' do + exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['{}']) + token = CountsExportDownload.prepare(exporter, 'owner') + expect(CountsExportDownload.find(token, 'owner')).not_to be_nil + expect(CountsExportDownload.find(token, 'other')).to be_nil + post = create(:post) + PostVersion.create!(post:, version_no: 1, event_type: 'discard', url: post.url, + title: 'private', tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024)) + expect(CountsExportDownload.find(token, 'owner')).to be_nil + end + + it 'rejects oversized output and removes its partial file' do + stub_const('CountsExportDownload::MAX_FILE_BYTES', 4) + exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['12345']) + expect { CountsExportDownload.prepare(exporter, 'owner') } + .to raise_error(CountsExportDownload::TooLarge) + expect(Dir.glob(CountsExportDownload.directory.join('counts-*'))).to be_empty + end + + it 'expires queued work after worker loss and rejects expired download signatures' do + CountsExportJobs.pulse + token = CountsExportJobs.start(payload, 'owner', 'ja') + travel_to(Time.current + CountsExportJobs::HEARTBEAT_SECONDS + 1) do + expect(CountsExportJobs.status(token, 'owner')).to include( + 'status' => 'failed', 'error' => 'worker_unavailable') + end + exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['{}']) + download = CountsExportDownload.prepare(exporter, 'owner') + travel_to(Time.current + CountsExportDownload::EXPIRY + 1) do + expect(CountsExportDownload.find(download, 'owner')).to be_nil + end + end +end + +RSpec.describe 'Counts download acknowledgement' do + around do |example| + Dir.mktmpdir('counts-ack-spec') do |path| + @counts_ack_dir = Pathname.new(path) + HistoryAccess.set(user: nil) { example.run } + end + end + + before do + allow(CountsExportDownload).to receive(:directory).and_return(@counts_ack_dir) + end + + it 'binds an acknowledgement to the owner, job, file token and fresh attempt' do + CountsExportJobs.pulse + payload = { 'resource' => 'posts', 'definition' => 'history', 'mode' => 'details', + 'format' => 'json', 'locale' => 'ja' } + token = CountsExportJobs.start(payload, 'owner', 'ja') + id = CountsExportJobs.authorised_id(token, 'owner') + state = CountsExportJobs.read(id).merge('status' => 'ready', 'token' => 'file-token') + CountsExportJobs.write(state) + receipt = CountsExportJobs.download_attempt(token, 'owner') + expect(receipt['download_status']).to eq('pending') + arguments = { file_token: 'file-token', attempt: receipt.fetch('download_attempt') } + expect(CountsExportJobs.download_attempt(token, 'other', **arguments)).to be_nil + expect(CountsExportJobs.download_attempt(token, 'owner', **arguments.merge(file_token: 'wrong'))).to be_nil + CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'streaming') + CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'started') + expect(CountsExportJobs.status(token, 'owner')['download_status']).to eq('started') + expect(CountsExportJobs.download_attempt(token, 'owner')['download_attempt']) + .to eq(receipt['download_attempt']) + CountsExportJobs.download_attempt(token, 'owner', **arguments, status: 'failed') + fresh = CountsExportJobs.download_attempt(token, 'owner') + expect(fresh['download_attempt']).not_to eq(receipt['download_attempt']) + expect(CountsExportJobs.download_attempt(token, 'owner', **arguments)).to be_nil + CountsExportJobs.status(token, 'owner', cancel: true) + expect(CountsExportJobs.download_attempt(token, 'owner')).to be_nil + end +end + +RSpec.describe CountsExportDownload::ResponseBody do + it 'survives Rails buffer enumeration without closing before the first byte' do + Dir.mktmpdir('counts-body-spec') do |directory| + path = File.join(directory, 'file') + payload = 'data' * 20_000 + File.binwrite(path, payload) + states = [] + source = ActionDispatch::Response::FileBody.new(path) + body = described_class.new(source) { |state| states << state; true } + response = ActionDispatch::Response.new + response.body = body + deferred = response.stream.to_ary + expect(states).to eq([]) + expect(deferred.to_a.join).to eq(payload) + expect(states.count('started')).to eq(1) + end + end + + it 'acknowledges only after the first Rack yield and closes its bounded file body' do + Dir.mktmpdir('counts-body-spec') do |directory| + path = File.join(directory, 'file') + File.binwrite(path, 'x' * (64 * 1024 + 3)) + states = [] + source = ActionDispatch::Response::FileBody.new(path) + body = described_class.new(source) { |state| states << state; true } + chunks = [] + body.each do |chunk| + expect(states).not_to include('started') if chunks.empty? + chunks << chunk + end + expect(chunks.join).to eq('x' * (64 * 1024 + 3)) + expect(chunks.map(&:bytesize).max).to be <= 64 * 1024 + expect(states.count('started')).to eq(1) + expect(body).not_to respond_to(:to_path) + expect { body.close }.not_to raise_error + end + end + + it 'records a failed stream without converting an exception into a second HTTP response' do + Dir.mktmpdir('counts-body-spec') do |directory| + path = File.join(directory, 'file') + File.binwrite(path, 'data') + states = [] + source = ActionDispatch::Response::FileBody.new(path) + body = described_class.new(source) { |state| states << state; true } + expect { body.each { raise IOError, 'connection closed' } }.to raise_error(IOError) + expect(states).to include('failed') + expect(states).not_to include('started') + expect { body.close }.not_to raise_error + end + end +end diff --git a/backend/spec/services/counts_pivot_security_spec.rb b/backend/spec/services/counts_pivot_security_spec.rb new file mode 100644 index 0000000..337d7a5 --- /dev/null +++ b/backend/spec/services/counts_pivot_security_spec.rb @@ -0,0 +1,197 @@ +require 'rails_helper' +require 'csv' + +RSpec.describe 'Counts masked pivot contracts' do + let(:admin) { create(:user, :admin) } + let(:member) { create(:user, :member) } + let(:guest) { create(:user, :guest) } + let(:params) do + { 'resource' => 'posts', 'definition' => 'history', 'mode' => 'summary', + 'locale' => 'ja', 'execution_at' => '2025-01-01T00:00:00Z', + 'timezone' => 'UTC', 'rows' => 'tags', 'cols' => 'event_type', 'sort' => '' } + end + let!(:tags) do + ['A', 'B', 'C'].map do |name| + tag = create(:tag, primary_name: name) + TagVersion.create!(tag:, version_no: 1, event_type: 'create', name:, + category: 'general', aliases: '', parent_tag_ids: '', + names: { ja: { name:, script_code: 'Jpan', auto_generated: false, aliases: [] } }, + created_at: Time.utc(2023)) + tag + end + end + + before do + [[0, 1], [0], [1, 2]].each_with_index do |indexes, index| + post = create(:post) + snapshot = indexes.map { { tag_id: tags[_1].id, version_no: 1, sections: [] } } + PostVersion.create!(post:, version_no: 1, event_type: 'create', + title: "hidden-title-#{ index }", url: post.url, tags: '', tags_json: snapshot, + parent_post_ids: '', created_by_user: member, created_at: Time.utc(2024, 1)) + next if index == 2 + + PostVersion.create!(post:, version_no: 2, event_type: 'discard', + title: "private-discard-#{ index }", url: post.url, tags: '', tags_json: [], + parent_post_ids: '', created_by_user: member, created_at: Time.utc(2024, 2)) + end + end + + def result(user, extra = {}) + HistoryAccess.set(user:) { CountsTableQuery.new(params.merge(extra)).call } + end + + def grand(result) + result.fetch(:pivot_totals).find { _1[:row].empty? && _1[:column].empty? }.fetch(:count) + end + + it 'preserves monthly name subtotals and grand totals across pages' do + settings = { 'resource' => 'tags', 'rows' => 'created_at:month name' } + complete = result(admin, settings) + expect(complete[:group_count]).to eq(3) + expect(grand(complete)).to eq(3) + (1..3).each do |page| + paged = result(admin, settings.merge('limit' => '1', 'page' => page.to_s)) + expect(grand(paged)).to eq(3) + subtotal = paged[:pivot_totals].find do |cell| + cell[:row] == ['2023-01'] && cell[:column].empty? + end + headers = paged[:row_headers].select { _1[:kind] == 'subtotal' } + if page == 3 + expect(headers.map { _1[:values] }).to eq([['2023-01']]) + expect(subtotal).not_to be_nil + expect(subtotal[:count]).to eq(3) + else + expect(headers).to eq([]) + expect(subtotal).to be_nil + end + expect(paged[:cells].sum { _1[:count] }).to eq(1) + expect(paged[:group_count]).to eq(3) + end + end + + it 'preserves five tag occurrences after merging private tags and across pages' do + [nil, guest, member, admin].each do |user| + page = result(user) + expect(grand(page)).to eq(5) + expect(page[:cells].sum { _1[:count] }).to eq(5) + expect(page).not_to have_key(:excluded_count) + paged = result(user, 'limit' => '1') + expect(grand(paged)).to eq(5) + expect(result(user, 'limit' => '1', 'page' => '2')[:group_count]) + .to eq(page[:group_count]) + end + public_page = result(nil) + hidden = public_page[:cells].select { _1[:values].first == CountsAxes::MASKED_VALUE } + expect(hidden.sum { _1[:count] }).to eq(3) + expect(public_page[:axis_labels]['a0']).not_to have_key("internal:#{ tags.first.id }") + expect(public_page.to_json).not_to include('hidden-title', 'private-discard') + end + + it 'retains distinct membership when tags are not a row axis' do + [nil, admin].each do |user| + expect(grand(result(user, 'rows' => 'event_type', 'cols' => 'tags'))).to eq(3) + expect(result(user, 'rows' => '', 'cols' => '')[:cells].sole[:count]).to eq(5) + details = result(user, 'mode' => 'details', 'rows' => '', 'cols' => '') + expect(details[:count]).to eq(user == admin ? 5 : 1) + end + end + + it 'uses changed-record membership for totals and hides changes to private predecessors' do + post = PostVersion.order(:id).last.post + PostVersion.create!(post:, version_no: 2, event_type: 'update', title: 'changed-public-title', + url: post.url, tags: '', + tags_json: [{ tag_id: tags.last.id, version_no: 1, sections: [] }], + parent_post_ids: '', created_at: Time.utc(2024, 3)) + restored = PostVersion.order(:id).first.post + PostVersion.create!(post: restored, version_no: 3, event_type: 'restore', + title: 'restored-public-title', url: restored.url, tags: '', + tags_json: [{ tag_id: tags.first.id, version_no: 1, sections: [] }], parent_post_ids: '', + created_at: Time.utc(2024, 4)) + [nil, admin].each do |user| + page = result(user, 'rows' => 'tags changed_field', 'cols' => 'event_type') + expect(grand(page)).to eq(page[:cells].sum { _1[:count] }) + expect(page[:cells]).not_to be_empty + if user.nil? + expect(page[:cells].any? { _1[:values][1] == CountsAxes::MASKED_VALUE }).to be(true) + end + expect(page.to_json).not_to include('hidden-title', 'private-discard') + end + end + + it 'counts distinct changed records when tags are not additive row tuples' do + [nil, admin].each do |user| + page = result(user, 'rows' => 'changed_field', 'cols' => 'event_type') + expect(grand(page)).to eq(2) + if user.nil? + expect(page[:cells].map { _1[:values].first }.uniq).to eq([CountsAxes::MASKED_VALUE]) + end + # Changed discard snapshots have no tags: no full cell contributes here. + empty = result(user, 'rows' => 'changed_field', 'cols' => 'tags') + expect(empty[:cells]).to be_empty + expect(grand(empty)).to eq(0) + end + end + + it 'keeps NULL as a public data value rather than treating it as a rollup' do + post = create(:post) + PostVersion.create!(post:, version_no: 1, event_type: 'create', title: 'public bot', + url: post.url, tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 5)) + page = result(nil, 'rows' => 'user', 'cols' => 'event_type') + expect(page[:cells].select { _1[:values].first.nil? }.sum { _1[:count] }).to eq(1) + expect(grand(page)).to eq(6) + expect(page[:row_headers]).to include(hash_including(values: [nil], kind: 'data')) + end + + it 'sorts public numeric values numerically alongside a masked group' do + post = PostVersion.order(:id).last.post + [2, 10].each do |number| + PostVersion.create!(post:, version_no: number, event_type: 'update', + title: 'public numeric sort', url: post.url, tags: '', tags_json: [], + parent_post_ids: '', created_at: Time.utc(2024, 5)) + end + page = result(nil, 'rows' => 'version_no', 'cols' => '') + values = page[:row_headers].select { _1[:kind] == 'data' }.map { _1[:values].first } + expect(values.reject { _1 == CountsAxes::MASKED_VALUE }.map(&:to_i)).to eq([1, 2, 10]) + expect(values.last).to eq(CountsAxes::MASKED_VALUE) + end + + it 'rejects private predicates and supplied measure coordinates before SQL labels escape' do + probes = [{ 'resource_id' => PostVersion.first.post_id.to_s }, + { 'tags' => 'A' }, + { 'conditions' => '', 'conditions.0.field' => 'title', 'conditions.0.operator' => 'eq', + 'conditions.0.value' => 'hidden-title-0' }, + { 'drill' => '', 'drill.0.field' => 'tags', 'drill.0.value_type' => 'string', + 'drill.0.value' => "internal:#{ tags.first.id }" }, + { 'cols' => 'tags', 'column_order' => '', 'column_order.0.value_type' => 'string', + 'column_order.0.value' => CountsAxes::MASKED_VALUE }] + HistoryAccess.set(user: nil) do + probes.each do |probe| + expect { CountsTableQuery.new(params.merge(probe)) } + .to raise_error(CountsAxes::PrivateCondition) + end + end + end + + it 'keeps public column sorting usable while guessed private coordinates match nothing' do + first = result(nil, 'rows' => 'event_type', 'cols' => 'tags') + probe = result(nil, 'rows' => 'event_type', 'cols' => 'tags', + 'column_order' => '', 'column_order.0.value_type' => 'string', + 'column_order.0.value' => "internal:#{ tags.first.id }") + expect(probe[:column_values]).to eq(first[:column_values]) + expect(probe[:cells]).to eq(first[:cells]) + expect(probe[:column_values_editable]).to be(false) + expect(probe.to_json).not_to include("internal:#{ tags.first.id }") + end + + it 'exports masked coordinates and safe CSV labels from the same pivot' do + HistoryAccess.set(user: nil) do + payload = params.merge('format' => 'json') + json = JSON.parse(CountsTableExport.new(payload).body.to_a.join) + expect(json['data'].sum { _1['kind'] == 'data' ? _1['count'] : 0 }).to eq(5) + expect(json.to_json).not_to include('hidden-title', 'private-discard') + csv = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join + expect(csv).to include('********') + expect(csv).not_to include(CountsAxes::MASKED_VALUE, 'hidden-title', 'private-discard') + end + end +end diff --git a/backend/spec/services/query_state_spec.rb b/backend/spec/services/query_state_spec.rb new file mode 100644 index 0000000..81f84bc --- /dev/null +++ b/backend/spec/services/query_state_spec.rb @@ -0,0 +1,33 @@ +require 'rails_helper' + +RSpec.describe QueryState do + it 'decodes typed axis values and keeps absence distinct from an empty set' do + { 'boolean' => ['false', false], 'number' => ['7', 7], + 'string' => ['a b & c', 'a b & c'], 'null' => ['', nil] }.each do |type, (raw, expected)| + params = { 'conditions' => '', 'conditions.0.field' => 'user', + 'conditions.0.operator' => 'axis_eq', + 'conditions.0.value_type' => type, 'conditions.0.value' => raw } + expect(described_class.records(params, 'conditions').sole['value']).to eq(expected) + end + expect(described_class.records({}, 'conditions')).to eq([]) + expect(described_class.records({ 'conditions' => '' }, 'conditions')).to eq([]) + expect(described_class.records({ 'conditions' => '', 'conditions.0.values' => '1' }, + 'conditions')).to eq([{ 'values' => true }]) + end + + it 'rejects gaps, unknown keys, JSON, wrong types and oversized lists' do + [ + { 'conditions' => '[]' }, { 'conditions' => '', 'conditions.1.field' => 'user' }, + { 'conditions' => '', 'conditions.0.sql' => 'TRUE' }, + { 'conditions' => '', 'conditions.0.blank' => 'true' }, + { 'conditions' => '', 'conditions.0.value' => ['x'] }, + { 'drill' => '', 'drill.0.field' => 'user', 'drill.0.value_type' => 'object' }, + { 'conditions' => 'x' * 131_073 }].each do |params| + expect { described_class.validate!(params, ['conditions', 'drill']) } + .to raise_error(ArgumentError) + end + ['user user', 'user title', ' user', (1..201).to_a.join(' ')].each do |value| + expect { described_class.list(value) }.to raise_error(ArgumentError) + end + end +end diff --git a/backend/spec/services/shared_resource_id_allocator_mysql_spec.rb b/backend/spec/services/shared_resource_id_allocator_mysql_spec.rb index e6b2aa4..8487c30 100644 --- a/backend/spec/services/shared_resource_id_allocator_mysql_spec.rb +++ b/backend/spec/services/shared_resource_id_allocator_mysql_spec.rb @@ -8,6 +8,10 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do expect(Rails.env.test?).to be(true) expect(ApplicationRecord.connection.adapter_name).to match(/mysql/i) expect(ApplicationRecord.connection_pool.size).to be >= 3 + @main_connection = ApplicationRecord.connection + # A leaked fixture pin makes worker threads share the main session. Fail before + # starting a worker rather than diagnosing IS_USED_LOCK(NULL) after a timeout. + expect(@main_connection.open_transactions).to eq(0) @prefix = "shared_ids_#{ SecureRandom.hex(8) }" @tag = create(:tag, primary_name: "#{ @prefix }_owner") @group = TagGroup.create!(name: "#{ @prefix }_owner", version_no: 1) @@ -42,13 +46,20 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do end def await_queue queue - Timeout.timeout(15) { queue.pop } + value = Timeout.timeout(15) { queue.pop } + raise value if value.is_a?(Exception) + + value end def worker &block Thread.new do Thread.current.report_on_exception = false - ApplicationRecord.connection_pool.with_connection(&block) + ApplicationRecord.connection_pool.with_connection do |connection| + raise 'Shared ID worker reused the main connection' if connection.equal?(@main_connection) + + block.call(connection) + end end.tap { @threads << _1 } end @@ -86,11 +97,16 @@ RSpec.describe 'Shared resource IDs across MySQL connections', type: :model do ActiveSupport::Notifications.subscribed(subscriber, 'sql.active_record') do id = create_resource(kind).id end + raise 'Shared ID creation did not acquire a named lock' unless lock_name + ready << [id, connection.select_value('SELECT CONNECTION_ID()').to_i, lock_name] await_queue(@release) raise ActiveRecord::Rollback if rollback end id + rescue StandardError => error + ready << error + raise end [thread, await_queue(ready)] end diff --git a/docs/counts-exports.md b/docs/counts-exports.md new file mode 100644 index 0000000..3d00934 --- /dev/null +++ b/docs/counts-exports.md @@ -0,0 +1,164 @@ +# Counts エクスポートワーカー + +Counts の画面からのエクスポートは永続ファイルキューへ登録する。 +Active Job の既定のプロセス内 async アダプターは使用しない。 +追加 gem、外部キュー、DB マイグレーションは不要。 + +## 起動・配置 + +backend で `bundle exec rake counts:export_worker` を、systemd 等の監視対象として +別プロセスで起動する。Web プロセス内では起動しない。 +1監視プロセスは同時に1ジョブを専用の子プロセスで生成し、全体の実行枠は +既存の2枠を共有する。fork 前に DB 接続を閉じ、子が独立した接続を取得する。 +Web 内のスレッドや Active Job async では実行しない。Linux / POSIX の fork が必要。 +監視側は SQL 中も1秒間隔でジョブとワーカーの heartbeat を更新する。 + +`docs/systemd/btrc-counts-exports.service` を配備用 unit として追加している。 +Restart=always、KillMode=control-group、停止猶予150秒を指定している。 +User、WorkingDirectory、bundle が見つかる PATH を実際の配備先へ合わせる。 +`/etc/btrc-hub/counts-exports.env` に Web と同じ Rails secret・接続設定・ +COUNTS_EXPORT_DIRECTORY を設定し、同じ Unix ユーザーで読み書きできるようにする。 +OS 起動時の有効化・サービス起動は運用側で行う必要がある。 +今回 unit のインストール、有効化、本番での起動・再起動は行っていない。 + +Web とワーカーは同一リリース、Rails secret、アプリケーション環境、DB 接続先を +使用する。DB コネクションはワーカープロセスごとに必要。 +`COUNTS_EXPORT_DIRECTORY` を同じ永続 POSIX 共有ディレクトリに設定する。 +省略時は `backend/tmp/counts_exports`。 +全プロセスで同一の所有者とアクセス権を使用する。 +複数ホストでは、flock・同一ファイルシステム内の atomic rename・fsync が正しく +機能する共有ボリュームが必須。オブジェクトストレージや各コンテナ独立の tmp は不可。 +ジョブ JSON と完成ファイルが格納されるため、このディレクトリを静的公開しない。 + +## API と進捗 + +- POST `/counts/export/jobs`：要求検証・実行時点の固定・署名付きジョブトークンの発行。 +- GET `/counts/export/jobs/:token`：状態、実際の処理済みレコード数、判明している総数。 +- DELETE `/counts/export/jobs/:token`：取消を永続記録。 +- POST `/counts/export/jobs/:token/download`：セッション所有者の開始確認要求を登録。 +- GET `/counts/export/files/:token?job=...&attempt=...`：実 GET の認可・送出開始の確認。 +- GET `/counts/export/files/:token`：既存クライアント向けの認可付きダウンロード。 + +ジョブ・ファイルのトークンは作成元のセッションに紐付く。 +既存の BAN・CSRF 制御を使用する。フロントは約800ms間隔で状態を取得する。 +明細の総数は同じ読み取りトランザクション内の SQL 件数を使用する。 +queued と running に加え、running 内の phase で aggregating（集計中）と +writing（出力中）を区別する。最初の SQL より前に aggregating を記録する。 +heartbeat_at は監視側の生存時刻、updated_at は生成処理が状態を書いた時刻であり、 +処理件数と分離する。SQL 中に件数は増やさない。 +集計の総出力数は生成終了まで未確定とし、不確定進捗として扱う。 +processed は CSV では実際に生成したデータ行数、JSON のクロス集計では +実際に生成した pivot_cell 数。ヘッダーは含めず、小計・合計の物理レコードは含む。 +通常の集計行だけを数える logical_count_unit=regular_result_rows と区別する。 +出力中は約250ms間隔と最終レコードで実測件数を保存する。 +完成状態はファイルの生成完了であり、ブラウザの保存完了を意味しない。 + +## 取消・障害・制限 + +通常は約250ms間隔の生成チェックポイントで取消を検知する。 +監視側も取消を確認し、子へ TERM、5秒後も終了していなければ KILL を送る。 +子の回収後に一時ファイルを削除する。DB 側の実行停止は切断検知等に依存するため、 +取消 API の受理と DB の即時停止は同義ではない。 +取消済みへの遷移は生成処理の終了と一時ファイル削除後に行う。 +既に始まった HTTP 転送やブラウザ側の保存は取り消せない。 + +ワーカーが異常終了したジョブは、次のワーカーが実行ロックを取得した際に失敗状態にし、 +そのジョブの残存生成ファイルを削除する。別の読み取り状態で自動再実行しない。 +原子的に書き換える状態ファイルとプロセス間ロックにより、二重処理・取消との競合を防ぐ。 + +既存の同時実行2、1ファイル128MiB、保存領域512MiB、生成120秒、期限10分を維持する。 +生成120秒は監視側でも強制する。ワーカー未起動なら登録 API は503を返す。 +待機中のワーカー消失、15秒以上の heartbeat 停止、150秒以上の待機は失敗とする。 +ネットワーク切断時も UI が無期限に処理中にならないよう、状態取得は1回15秒で中断し、 +30秒以上状態を取得できなければ失敗表示へ移る。処理件数を捏造しない。 +画面からの通信断だけではサーバーの取消完了を保証しない。サーバー側の期限は独立して働く。 +期限切れのジョブ・ファイル・残存状態書込ファイルは登録時またはワーカーの巡回時に削除する。 +ワーカーと Web の両方を停止した間は巡回削除も停止する。 + +これらの処理は静的確認のみで、実環境のキュー処理・取消・転送・共有ボリュームでの +排他動作は未確認。 + +## Counts の件数と性能 + +通常の Counts 応答では、使用していない全履歴の population_count は計算しない。 +キーは null、population_count_computed は false とし、0に置き換えない。 +summary は非公開履歴を含む同じ母集団を集計する。非公開の軸はサーバーの CASE 式で +単一の予約値へまとめてから GROUP BY し、output_count / visible_output_count はその +安全化した通常行数を表す。excluded_count は返さない。details は従来どおり +output_count が権限適用前、visible_output_count が適用後の明細行数である。 +小計・合計の物理行は行数に含めない。 +CSV の物理行と JSON の pivot_cell には集計レコードを含むため、両者を混同しない。 + +除票境界・参照禁止集合は GROUP BY で識別キー単位にまとめる。 +投稿の参照判定は、全除票履歴を JSON_CONTAINS / FIND_IN_SET で反復照合せず、 +保存済みスナップショットから参照IDを JSON_TABLE で展開して照合する。 +権限条件付きの母集団件数は同一 export インスタンス内で再利用する。 +多段小計は、全軸の JOIN へ実際に寄与するレコードを CTE に保持し、各枝で同じ +母集合を使う。changed_field には直前版がある実変更だけが寄与する。タグが行軸に +ない場合は DISTINCT 投稿版を数える。タグが行軸にある場合は投稿版×タグを数え、 +全行軸の葉単位から小計・合計を求める。変更項目も行軸ならその寄与も保持する。 +private のタグ・変更項目をまとめても、内部の寄与は捨てず、ラベルだけを保護する。 + +スキーマ上、以下の追加インデックスが候補となる。migration は作成・実行していない。 + +- post_versions: (event_type, post_id, version_no) +- tag_versions / nico_tag_versions: (event_type, tag_id, version_no) +- tag_description_versions: (event_type, tag_id, language_code, version_no) +- tag_group_versions: (event_type, tag_group_id, version_no) +- material_versions: (event_type, material_id, version_no) +- 各履歴テーブル: (event_type, id)。ファイル認可の policy revision 用 MAX(id) の候補。 +- post_versions 等で未整備の (created_at)。履歴の異動期間を絞る場合の候補。 +- post_versions: (post_id, created_at, version_no)。日時指定の最新版探索の候補。 + +追加は許可後の新規 add_index migration として行い、既存 migration は変更しない。 +実DBでのインデックス容量・作成時ロック・実行計画・時間は未確認。 +索引追加なしでの最大データ量に対する応答時間は保証しない。 + + +## GET 応答開始の確認（静的実装・未実測） + +進捗画面は ready 後に owner/job に紐づく download_attempt を登録し、同じ attempt を +native iframe の実 GET に渡す。サーバーは署名・所有者・権限 revision・期限・ファイルを +再検証し、64KiB の最初のチャンクを Rack 出力側へ渡した後に started を記録する。 +HEAD や受付 POST では started にしない。確認には30秒の独立した期限がある。 +サーバーの出力開始はブラウザー受信・保存完了を保証しない。途中の切断もあり得る。 +開始確認前の失敗はダイアログに残る。再実行はダイアログを閉じてエクスポートをやり直す。 + +Web と worker は同じ COUNTS_EXPORT_DIRECTORY、Rails secret、DB、実行ユーザーで +共有 POSIX filesystem を使うこと。別ホストではローカル tmp を共有できない。 +systemd の EnvironmentFile と Web の環境を照合し、directory の owner と0700相当の +アクセス制限、容量、期限切れ削除、worker heartbeat と restart を公開前に確認する。 +この作業では systemctl・worker 起動・本番環境への適用を行っていない。 + +これらのコード対策と回帰テストを追加しているが、テスト・型チェック・build は未実行。 +middleware/proxy 経由の転送は別途実環境で確認する。 + + +## 開始確認の競合と権限 + +開始受付は pending、実 GET の単独取得は streaming、最初の Rack yield 後は started。 +同じ job の重複受付は既存 attempt を返す。streaming への遷移はプロセス間ロックで +単独取得し、同一 attempt の二重 GET を拒否する。開始前の attempt は45秒で失効する。 +Frontend は30秒の独立タイマーで確認を打ち切る。一時的なポーリング通信断は同じ +attempt で再取得するが、別の GET は始めない。取消失敗後の再試行は利用者の再操作で行う。 + +ファイル・ジョブは owner、署名、期限、現在の権限 revision 及び artifact_policy を +検証する。最初の送出前にも現ユーザー・BAN・ファイル権限を再確認する。旧ポリシーの +生成物は再利用しない。送出中は各チャンクで取消状態を確認し、例外時はログと failed +状態を記録して body を閉じる。コミット後に head で別レスポンスを書かない。 + +ResponseBody に to_path を提供せず、sendfile middleware による ACK の迂回を避ける。 +Last-Modified と X-Accel-Buffering: no を設定するが、プロキシの動作まで保証しない。 +サーバーの最初の Rack yield は、クライアントの受信開始の厳密な証明ではない。 + +非管理者の summary では、公開集計属性は操作日時・異動区分。その他の属性に対する +抽出条件・補助タグ検索は拒否する。measures / column_order はマスク後の式だけと +比較するため、推測した private ID は一致しない。予約値そのものの条件指定は拒否する。 +private ID の正誤に応じて許可可否を変えないため、条件の一致探索を避けられる。 +非公開列を含む列順変更は API の column_values_editable でも不可と通知する。 +公開列の値を使うソートは引き続き利用できる。 +明細には元の認可を適用し、抽出条件自体や取得済みの全言語値を表示設定で改変しない。 +外部タグ履歴の linked_tags は名前しか記録していないため、非公開タグの記録名を含む +スナップショットを保守的に非公開とする。名前全体を空白区切りの境界で照合する。 +区切りが曖昧な旧データでは過剰な秘匿の可能性はあるが、参照 ID がない旧形式から +名前の公開可否を楽観的に推定しない。 diff --git a/docs/counts-language-columns.md b/docs/counts-language-columns.md new file mode 100644 index 0000000..3b4f45c --- /dev/null +++ b/docs/counts-language-columns.md @@ -0,0 +1,26 @@ +# Counts の言語別列 + +- 列は `languages.created_at ASC, languages.code ASC` の順に生成する。 + 全名称列の後に全別名列を置く。Locale の数によって列を重複させない。 +- 識別子は `name_ja`、`name_ain`、`aliases_ja`、`aliases_ain` 等とする。 + 言語の名称は Frontend の common 辞書を使う。 +- 名称は記録済みの言語の主名称、別名はその言語の全別名を使う。 + 特定の文字体系で絞らず、記録済みの script_code 等を保持する。 + 履歴の時点評価と HistoryAccess は従来どおり適用する。 +- 初期選択には UI ロカールの言語コードを使う。明示的な列・空選択は上書きしない。 + +## 旧 URL・API + +- 出力列、子列、列順、ソートの旧 `name_ain_kana` 等は、入力境界で + `name_ain` 等へ正規化する。表示・ソートは言語全体の値を対象とするため、 + 従来表示されなかった文字体系の値が表示され、並びが変わる場合がある。 +- 抽出条件の旧ロカール付きキーは別扱いとし、従来の文字体系限定 SQL を維持する。 + 新規条件候補には出さない。既存条件の表示・編輯時だけ文字体系付きの項目名を示す。 + 利用者が新しい言語単位の項目を選択した場合にだけ、その条件を言語全体へ切り替える。 +- 新しい条件の解釈を `name_scope=language` で明示する。 + 未指定又は `locale` の旧条件は旧仕様として扱い、例えば `name_ja` を + `name_ja_jpan` という入力互換専用キーへ変換する。これにより Jpan 限定を保つ。 + 正規化後の URL は `name_scope=language` と互換専用キーを保存し、再解釈を防ぐ。 +- 新しい API 利用者が言語単位の条件を送る場合も `name_scope=language` を指定する。 + 指定のない既存 API の `name_ja` / `aliases_ja` 条件は従来の Jpan 限定として扱う。 + 未知の name_scope は 400 とする。JSON・Base64 をクエリに格納しない。 diff --git a/docs/counts-release-checklist.md b/docs/counts-release-checklist.md new file mode 100644 index 0000000..d0a7312 --- /dev/null +++ b/docs/counts-release-checklist.md @@ -0,0 +1,148 @@ +# Counts リリース前確認 + +この作業では Git、テスト、lint、typecheck、build、migration、サーバー・worker・ +systemd の起動、本番適用を行っていない。コードの静的確認とテストコードの編集のみ。 +以下の実行と結果確認は利用者が行う。未実行の状態は GO としない。 + +## 対応箇所 + +- 全件集計と安全な軸：counts_axes.rb / counts_table_query.rb / counts_pivot_totals.rb。 + 非公開値は GROUP BY 前に統合。通常セル・ソート用集計・小計は同じ寄与母集合を使う。 +- 出力：counts_table_export.rb / countsDisplay.ts / CountsSummaryResults.tsx。 + CSV とコピーは伏字。JSON の非公開座標は予約値のみ。非公開セルはリンクを作らない。 +- 配布：counts_exports_controller.rb / counts_export_jobs.rb / counts_export_download.rb。 + owner/job/attempt、単独 GET、期限・現在権限・旧 artifact policy の拒否を確認する。 +- 表示：summary 行番号を廃止し「行数」を表示。details の行番号・件数は維持。 + 非公開列座標のソート・並び替え可否はサーバーの能力フラグへ従う。 +- テスト基盤：requestAnimationFrame の同期・時刻0固定を修正。JSDOM の Canvas は + 計測非対応を返すテスト専用モックを置く。実ブラウザーの計測動作は変更していない。 + +## 利用者によるテスト + +backend ディレクトリで提案する対象（未実行）： + +```sh +bundle exec rspec spec/requests/counts_table_spec.rb spec/requests/counts_download_spec.rb spec/services/counts_pivot_security_spec.rb spec/services/counts_export_spec.rb spec/models/history_access_spec.rb spec/requests/post_locale_filter_identity_spec.rb spec/requests/tag_description_wiki_spec.rb +``` + +frontend ディレクトリで提案する対象（未実行）： + +```sh +npm run test:run -- src/lib/counts.test.ts src/lib/api.test.ts src/pages/counts/CountsResourcePage.test.tsx src/pages/help/HelpChangesPage.test.tsx src/components/counts/CountsControls.test.tsx src/components/counts/CountsSummaryResults.test.tsx src/components/counts/CountsExportProgress.test.tsx +``` + +SSR は frontend/server.ssr.test.mjs が dist/server/entry-server.js を読み込む。 +ソースの PageHead はサイト名付きの共通タイトルを使用している。古い成果物のまま +SSR テストを実行しても現行ソースの検証にならない。利用者が最新成果物を用意してから +`npm run test:ssr` を実行する。Codex はビルド・SSR テストとも実行していない。 +主要テストの後に既存の全テスト群でも回帰の有無を確認する。 + +## ブラウザーと本番切替前 + +- anonymous / guest / member / admin で同じ期間・公開条件を指定し、通常セル・総計の + 数値を比較する。安全なグループ数は権限で異なり得る。details は秘匿版を出さない。 +- 投稿 A,B / A / B,C のタグ行軸は5。タグを行軸から外した合計は distinct 投稿数。 + changed_field、NULL、複数段小計、ページ境界、CSV / JSON でも値を比較する。 +- 伏字セル、API 辞書、DOM、コピー、CSV / JSON に非公開の値・ID がないこと。 + 非公開セルのドリルダウンはなく、private 条件の summary は安全な403となること。 +- 8軸・100列の専用エラー、SQL timeout、worker 不在、認証・権限、取消失敗を確認する。 +- CSV / JSON とも ready / 100% だけでは閉じず、実 GET の matching attempt が + started となってから閉じること。通信遮断・取消・再描画・タイムアウトも確認する。 + サーバー送出開始はブラウザーの受信完了・保存完了を保証しない。 +- PC・スマホで独立／共通スクロール、D&D、列幅、セル選択・文字列選択・キャレット、 + 内表 JSON の折返し、ページタイトル、Counts の noindex・先読み禁止を確認する。 + locale 切替後も対象 ID と集計値は同一で、全言語の登録値を保持すること。 +- Web と worker の同一リリース・secret・DB・COUNTS_EXPORT_DIRECTORY、所有者、 + flock / atomic rename、容量、期限切れ掃除、heartbeat、ログ、再起動を照合する。 +- systemd unit の User / WorkingDirectory / EnvironmentFile / PATH を実環境と照合する。 + 設定適用・サービス起動・本番切替はこの作業では行っていない。 +- ロール変更・logout・新しい discard 後に、生成済みファイルが再取得できないこと。 +- ロールバック時も公開ダウンロードを止め、旧生成物を再配布しない。古いコードへ戻すと + artifact_policy やマスクの保護を失う可能性があるため、無条件に巻き戻さない。 + +P0 の回帰テストと利用者の動作確認が済むまで判定は NO-GO。 + +## 静的な対応状況 + +| 対象 | 状態 | 根拠 | +| --- | --- | --- | +| 非公開軸・参照・全件母集団 | 対応済み（静的確認のみ） | HistoryAccess / CountsAxes / CountsTableQuery と回帰コード | +| タグ・変更項目・小計 | 対応済み（静的確認のみ） | CountsPivotTotals と全セル・ソートの母集合の統一 | +| API / CSV / JSON / コピー / URL | 対応済み（静的確認のみ） | マスク後の値のみを表示・比較し、予約座標の条件化を拒否 | +| GET ACK・競合・取消・期限 | 対応済み（静的確認のみ） | サーバー状態遷移、単独 GET、独立タイマー、回帰コード | +| P1・既知テストのソース上の問題 | 対応済み（静的確認のみ） | 行番号、行数、エラー、RAF、fixture、title 契約 | +| テスト成功・実ブラウザー・本番稼働 | 判定不能 | 実行禁止のため未検証 | + +## 今回の編集ファイル + +すべて Counts 本体に必要な変更。共有箇所の例外は HistoryAccess の参照秘匿と、 +api.ts の Abort/CSRF 待機・native download の機構であり、全域の安全性にも必要。 +SQL の速度改善・索引追加は別 Issue とし、今回 migration は追加していない。 + +### Backend App + +- backend/app/models/history_access.rb：外部タグの記録名参照と、投稿の記録済み参照版にも + 同じ可視性判定を適用。旧 artifact policy の生成物を拒否。 +- backend/app/services/counts_population.rb：軸条件の型・予約値の検証。 +- backend/app/services/counts_axes.rb：マスク前の内部寄与を計数し、直前版が非公開の場合も + changed_field の名前を秘匿。 +- backend/app/services/counts_table_query.rb：通常セル・ソートの寄与母集合、整数順、 + 条件表示名、公開列座標だけの操作、summary メタデータを整合。 +- backend/app/services/counts_pivot_totals.rb：全セルに寄与する母集合と、タグ行軸の加算。 +- backend/app/services/counts_table_export.rb：CSV の予約値を伏字化。 +- backend/app/services/counts_export_jobs.rb：開始確認の排他・状態遷移・失効と安全なエラーコード。 +- backend/app/services/counts_export_download.rb：64KiB の Rack body、開始確認、取消・close。 +- backend/app/controllers/counts_exports_controller.rb：GET 直前・送出直前の再認可と専用エラー。 + +### Frontend App + +- frontend/src/lib/api.ts：CSRF 待機中も中断可能にし、iframe の寿命・失敗通知を整理。 +- frontend/src/lib/counts.ts：マスク予約値と列操作の API 型。 +- frontend/src/lib/countsDisplay.ts：マスク表示と既知エラーの i18n 解決。 +- frontend/src/components/counts/CountsExportProgress.tsx：一つのジョブを維持し、matching ACK、 + 独立期限、取消・再描画・通信断を処理。保存完了の成功表示はしない。 +- frontend/src/components/counts/CountsSummaryResults.tsx：行番号コードの撤去、私的セルの + ドリルダウン抑止、公開列のソート維持。 +- frontend/src/components/counts/CountsSortHeader.tsx：禁止座標のソート抑止と伏字の理由表示。 +- frontend/src/pages/counts/CountsResourcePage.tsx：専用エラーと、初回失敗後にも再実行できる状態。 +- frontend/src/locales/ja/counts.json：承認されたエラー文言。 +- frontend/src/locales/ain-Kana/counts.json：同じキーと未翻訳・日本語 fallback の管理。 + +### テスト・テスト専用基盤 + +- backend/spec/services/counts_pivot_security_spec.rb（新規）：4権限、タグ寄与5、distinct、 + changed_field、private group、NULL、整数順、ページ、座標探索、JSON/CSV。 +- backend/spec/requests/counts_download_spec.rb（新規）：CSV/JSON の実 GET、HEAD、二重取得、 + 別セッション、失効、取消、ファイル不在、新しい discard。 +- backend/spec/services/counts_export_spec.rb：owner/job/attempt の ACK と Rack body の失敗。 +- backend/spec/models/history_access_spec.rb：旧 policy、外部タグ参照、記録済みタグ参照版。 +- backend/spec/requests/counts_table_spec.rb：専用軸数エラー。 +- frontend/src/components/counts/CountsExportProgress.test.tsx（新規）：matching ACK、StrictMode、 + 再描画、開始受付失敗、通信断、独立期限、取消の競合・再試行、unmount、iframe 失敗。 +- frontend/src/components/counts/CountsSummaryResults.test.tsx（新規）：伏字・理由・コピー・旧行番号。 +- frontend/src/components/counts/CountsControls.test.tsx：summary 行番号廃止と表示件数選択。 +- frontend/src/pages/counts/CountsResourcePage.test.tsx：初期エラーからの再実行、サイト名付き title。 +- frontend/src/lib/api.test.ts：共有 CSRF 待機中の中断、iframe の解放・重複失敗通知。 +- frontend/src/lib/counts.test.ts：理由別エラーと任意のサーバー文字列を表示しないこと。 +- frontend/src/test/setup.ts：時刻が進む非同期 RAF と Canvas のテスト専用 fallback。 +- frontend/server.ssr.test.mjs：共通 title の既存 assertion を維持し canonical と秘匿を補足。 + +### 文書 + +- docs/counts-exports.md：実装と一致する計数・ACK・権限・運用上の限界。 +- docs/counts-release-checklist.md：この一覧と利用者の検証・公開判定手順。 + +## 既知失敗の静的な確認結果 + +- counts_table_spec.rb：TagVersion.names の auto_generated、script_code、aliases があることを + 確認。新しい投稿タグ fixture にも schema 必須の sections を設定。 +- post_locale_filter_identity_spec.rb：検索文字列で基準集合を取得した後、tag_filters の + 応答から references を定義して構造化 URL を作る修正を維持。 +- tag_description_wiki_spec.rb：first / last とも対象 ID に限定した delete_all を維持。 +- CountsResourcePage.test.tsx / HelpChangesPage.test.tsx：共有 RAF の同期・時刻0固定を修正。 + Help の辞書応答は実 getHelpChanges が language を付ける契約も照合済み。 +- SSR title：現行ソースはサイト名を含む。dist を使うテストの assertion は削除していない。 + +上記はソース照合の結果であり、過去の失敗件数が0になったという実行結果ではない。 +既知の未修正 P0 を残して GO とする判断はしていない。現時点の阻害事項は、ユーザーに +よる MySQL・ブラウザー・worker/proxy を含む実検証がまだ行われていないことである。 diff --git a/docs/systemd/btrc-counts-exports.service b/docs/systemd/btrc-counts-exports.service new file mode 100644 index 0000000..9a879b5 --- /dev/null +++ b/docs/systemd/btrc-counts-exports.service @@ -0,0 +1,21 @@ +[Unit] +Description=BTRC Hub Counts export supervisor +Wants=network-online.target +After=network-online.target +StartLimitIntervalSec=0 + +[Service] +Type=simple +User=miteruzo +WorkingDirectory=/home/miteruzo/projects/btrc-hub/backend +Environment=RAILS_ENV=production +EnvironmentFile=/etc/btrc-hub/counts-exports.env +ExecStart=/usr/bin/env bundle exec rake counts:export_worker +Restart=always +RestartSec=3 +KillMode=control-group +TimeoutStopSec=150 +UMask=0077 + +[Install] +WantedBy=multi-user.target diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index ae68412..8431fdc 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -1,5 +1,41 @@ # frontend/AGENTS.md +## Mandatory security and URL contracts + +- URL クエリへの JSON 格納を禁止する。Base64・16進数等による迂回も禁止。 + 単純な配列は半角スペース区切り、構造化条件は番号付きの個別キーで表す。 + 未指定と明示的な空選択を区別し、自由入力を区切り文字で分割しない。 + Frontend / Backend で同じ文法・上限・検証を使い、新しい URL 状態の追加前に確認する。 +- 一度除票されたリソースの除票以前（除票版を含む）の履歴は admin 以外へ公開しない。 + 恢復しても解除しない。すべての後続除票を確定版番号と正しい複合キーで判定する。 + 本文・属性・ID・版番号・関連参照・旧値を API / SSR / キャッシュ / 出力から漏らさない。 + サーバーが確定した current_user の role のみを認可に使用する。 + 秘匿版も安全な全体件数には含めるが、公開明細・内訳・検索件数とは分離する。 + Counts では、現在の条件に対する権限除外前の出力行数と権限除外レコード数の + 数値公開及びその差分による間接的な情報開示を許容する。これは件数だけの例外であり、 + 秘匿された本文・属性・識別情報・集計ラベルを公開する例外ではない。 + それ以外の経路で秘匿属性を条件・軸・ソート・差分から推測できる実装は禁止する。 +- 秘匿は React / CSS では実装しない。認可済み API のみを使う。 + 認証・権限変更時は取得中リクエストと履歴キャッシュを破棄し、SSR の管理者データを共有しない。 + +## Mandatory Counts data contract + +- Counts のロカール非依存・内表行集合保持はルート AGENTS.md の必須契約に従う。 + 内表行の表示フィルターを作らない。名称・別名列はサーバーの Language コードを使い、 + 未登録列に別言語名を代入しない。列を隠しても取得済みの全言語データは保持する。 + ソート・コピー・JSON の投影には安定したキーと値を使い、UI 訳語を識別に使わない。 + +## Mandatory naming contract + +- ルート `AGENTS.md` の必須命名規約を適用する。非 callable の自前識別子に + 動詞を使わず、形容詞の状態名、名詞の真偽値には `Flg` を使用する。 + 例：`isLatest` は `latest`、`isAdmin` は `adminFlg`。 +- React / TanStack Query 等の外部 API 名は維持する。取得した値を自前の状態に + 保持するときは、必要に応じて分割代入等の境界で別名を付ける。 + callable の state setter、event handler、callback は動詞名を使用できる。 +- 型、表示項目キー、URL、API、コピー・エクスポートのフィールド参照を整合させる。 + 旧キーの互換性は共通の入出力境界で処理し、新しい内部状態には旧名を使用しない。 + ## Scope These rules apply to work under `frontend/`. @@ -33,9 +69,9 @@ npm run lint If either command cannot be run or fails, report the exact command and failure. -Do not create, modify, or run tests unless the user explicitly asks for test -work. When the user asks for tests, keep working and rerun them until they -pass or the remaining failure is clearly blocked. +明示的な個別指示がない限り、テストの追加・修正・削除・実行を一切行わない。 +fixture、mock、snapshot、テスト補助コード及び既存のテスト差分にも触れない。 +ビルド・lint・静的確認を使い、未確認事項はそのまま報告する。 ## TypeScript @@ -128,6 +164,19 @@ pass or the remaining failure is clearly blocked. ## Tailwind and UI +- 原則として、ユーザ向け UI 文言を TSX / TypeScript に直接記述しない。 + 新しい文言は既存の i18n システムで管理し、適切な既存キーを先に確認する。 + 不必要な重複定義を避け、新規機能固有の文言は適切な namespace にまとめる。 +- 対応 locale の辞書を整合させる。未翻訳項目には既存の日本語 fallback と + `_untranslated` 規約を使い、language metadata を維持する。 + 翻訳を捏造せず、未翻訳文字列を正式な翻訳として登録しない。 +- UI 文言の追加・変更時はビルド、静的検査、実画面の確認等で、辞書と + SSR / hydration の整合及び翻訳キーの非露出を検証する。 + i18n の検証を理由にテスト関連作業を開始しない。 +- API の内部識別子、パラメータ、ユーザ入力データ、外部 URL は機械的に + 翻訳しない。ボタン、項目名、CSV ヘッダ、エラー、空状態、ツールチップ、 + アクセシビリティ用ラベルも UI 文言として扱う。 + - Tailwind scans `src/**/*.{html,js,ts,jsx,tsx,mdx}`. - Use `cn` from `src/lib/utils.ts` for conditional class names and class merging. - In JavaScript, JSX, TypeScript, and TSX, use `cn` from `@/lib/utils` @@ -144,7 +193,7 @@ pass or the remaining failure is clearly blocked. heavy borders, or nested panel decoration unless the surrounding screen already uses them. - Keep operational screens dense and direct; trim explanatory copy and use - short Japanese labels that fit the control. + short localised labels that fit the control. - Preserve existing Japanese tone and orthography in nearby UI text, including old-kana wording where the file already uses it. - Do not add user-facing copy, helper text, descriptions, notes, tooltips, diff --git a/frontend/package-lock.json b/frontend/package-lock.json index 548c439..7fba620 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -11,6 +11,7 @@ "dependencies": { "@dnd-kit/core": "^6.3.1", "@dnd-kit/modifiers": "^9.0.0", + "@dnd-kit/sortable": "^10.0.0", "@dnd-kit/utilities": "^3.2.2", "@fontsource-variable/noto-sans-jp": "^5.2.9", "@mdx-js/react": "^3.1.1", @@ -574,6 +575,20 @@ "react": ">=16.8.0" } }, + "node_modules/@dnd-kit/sortable": { + "version": "10.0.0", + "resolved": "https://registry.npmjs.org/@dnd-kit/sortable/-/sortable-10.0.0.tgz", + "integrity": "sha512-+xqhmIIzvAYMGfBYYnbKuNicfSsk4RksY2XdmJhT+HAC01nix6fHCztU68jooFiMUB01Ky3F0FyOvhG/BZrWkg==", + "license": "MIT", + "dependencies": { + "@dnd-kit/utilities": "^3.2.2", + "tslib": "^2.0.0" + }, + "peerDependencies": { + "@dnd-kit/core": "^6.3.0", + "react": ">=16.8.0" + } + }, "node_modules/@dnd-kit/utilities": { "version": "3.2.2", "resolved": "https://registry.npmjs.org/@dnd-kit/utilities/-/utilities-3.2.2.tgz", diff --git a/frontend/package.json b/frontend/package.json index 7d93a4d..e9be868 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -18,6 +18,7 @@ "dependencies": { "@dnd-kit/core": "^6.3.1", "@dnd-kit/modifiers": "^9.0.0", + "@dnd-kit/sortable": "^10.0.0", "@dnd-kit/utilities": "^3.2.2", "@fontsource-variable/noto-sans-jp": "^5.2.9", "@mdx-js/react": "^3.1.1", diff --git a/frontend/server.mjs b/frontend/server.mjs index 9dfcc76..0e4a1e5 100644 --- a/frontend/server.mjs +++ b/frontend/server.mjs @@ -5,7 +5,7 @@ import { request as requestHTTPS } from 'node:https' import path from 'node:path' import { fileURLToPath } from 'node:url' -import { render, renderSitemap } from './dist/server/entry-server.js' +import { isCountsPath, render, renderSitemap } from './dist/server/entry-server.js' const root = fileURLToPath (new URL ('./dist/client/', import.meta.url)) const template = await readFile (path.join (root, 'index.html'), 'utf8') @@ -61,6 +61,8 @@ const server = createServer (async (request, response) => { { const url = new URL (request.url, origin) const pathname = decodeURIComponent (url.pathname) + const countsPage = isCountsPath (pathname) + if (countsPage) response.setHeader ('X-Robots-Tag', 'noindex') if (pathname === '/sitemap.xml' || pathname.startsWith ('/sitemap-')) { const result = await renderSitemap (url.pathname, origin, apiBaseURL) @@ -108,7 +110,7 @@ const server = createServer (async (request, response) => { return } const html = result?.html == null - ? template.replace ('', result?.status === 404 + ? template.replace ('', (result?.status === 404 || countsPage) ? '' : '') : template .replace ('lang="ja"', `lang="${result.locale}"`) diff --git a/frontend/server.ssr.test.mjs b/frontend/server.ssr.test.mjs index 668d111..cae7986 100644 --- a/frontend/server.ssr.test.mjs +++ b/frontend/server.ssr.test.mjs @@ -62,6 +62,38 @@ const responseFor = url => { return { locales: [{ code: 'ja', name: '日本語' }, { code: 'ain-Kana', name: 'アイヌ' }] } if (path === '/users/settings') return { theme: 'system', wiki_editor_mode: 'split' } if (path === '/users/theme_slots') return [] + if (path.startsWith ('/counts/definitions/')) { + const resource = path.split ('/')[3] + const language = locale === 'ain-Kana' ? 'ain' : 'ja' + const names = ['ja', 'ain'].map (code => ({ key: `name_${code}`, + label: 'counts:localeName', name_locale: { code, language_code: code } })) + const columns = resource === 'posts' ? ['resource_id', 'title'] + : ['resource_id', `name_${language}`] + return { fields: [{ key: 'resource_id', type: 'integer', nullable: false, + operators: ['eq', 'ne'] }], + output_fields: [{ key: 'resource_id', label: 'counts:columns.resourceId' }, + ...(resource === 'posts' ? [{ key: 'title', label: 'counts:columns.postTitle' }] : names)], + name_columns: names, filter_fields: [], column_aliases: [], condition_aliases: [], + axes: { latest: ['resource_id'], history: ['resource_id'] }, grains: ['month'], + defaults: [ + { definition: 'latest', mode: 'summary', params: [ + { key: 'rows', value: '' }, { key: 'cols', value: '' }] }, + ...['latest', 'history'].map (definition => ({ definition, mode: 'details', params: [ + { key: 'columns', value: columns.join (' ') }, + { key: 'sort', value: 'resource_id:asc version_no:asc' }] }))] } + } + if (path === '/counts/query') { + const header = { values: [], kind: 'data', depth: 0, axis_count: 0, label_key: null } + return { count: 1, page: 1, limit: 100, group_count: 1, + output_count: 1, visible_output_count: 1, excluded_count: 0, + excluded_count_scope: 'current_query', execution_at: date, + rows: [], cols: [], row_headers: [header], column_headers: [header], + column_values: [[]], cells: [{ values: [], count: 1 }], pivot_totals: [], + condition_labels: {}, axis_labels: {}, versions: [{ id: 1, resource_id: 1, + version_no: 1, latest: true, created_at: date, created_by_user: null, + title: 'SSR count', name_ja: 'SSR count', name_ain: 'SSR count', + names: [], aliases: [], parents: [], tags: [] }] } + } if (path === '/posts') return { posts: [post (locale)], count: 1 } if (path === '/gekanator/posts') return { posts: [post (locale)] } if (path === '/gekanator/questions') return { questions: [] } @@ -300,6 +332,40 @@ after (async () => { if (errors.length) throw new AggregateError (errors, 'SSR teardown failed') }) +test ('counts SSR renders the shell without result requests or embedded results', async () => { + for (const locale of ['ja', 'ain-Kana']) { + for (const [resource, mode] of ['posts', 'tags'].flatMap (resource => + ['summary', 'details'].map (mode => [resource, mode]))) { + const column = resource === 'posts' ? 'title' : (locale === 'ja' ? 'name_ja' : 'name_ain') + const query = + `definition=history&mode=${mode}&columns=resource_id+${ column }&user_id=4&timezone=UTC` + const start = requests.length + const result = await renderPage (`/${locale}/counts/${resource}?${query}`, origin, apiURL) + const calls = requests.slice (start).map (request => new URL (request.url, origin)) + assert.equal (result.status, 200) + assert.doesNotMatch (result.html, /SSR count|counts-grid|counts-nested/) + assert.match (result.html, /counts-targets/) + assert.match (result.html, /]/) + assert.doesNotMatch (result.html, /counts:|columns\.changedAt|fields\.actor/) + assert.match (result.head, /]*name="robots"[^>]*content="noindex"/) + const titles = { + ja: { posts: '広場集計', tags: 'タグ集計' }, + 'ain-Kana': { posts: 'ミンタㇻ　ウコピㇱキ', tags: 'シロシ　ウコピㇱキ' } } + assert.match (result.head, new RegExp ( + `]*>${titles[locale][resource]} \\|[^<]*<\/title>`)) + assert.match (result.head, /rel="canonical"/) + assert.doesNotMatch (result.html, /private-discard|hidden-title|__counts_private__/) + + assert.ok (calls.some (url => url.pathname === `/counts/definitions/${resource}`)) + assert.equal (calls.filter (url => url.pathname === '/counts/query').length, 0) + assert.doesNotMatch (result.bootstrap, /SSR count|conditionLabels|pivotTotals/) + const bootstrap = JSON.parse (result.bootstrap) + assert.equal (bootstrap.state.queries.filter (query => + query.queryKey[0] === 'counts' && query.queryKey[1] === resource).length, 0) + } + } +}) + test ('every canonical screen route returns page HTML without JavaScript', async t => { const cookie = await authenticatedCookie () for (const route of routes) { @@ -582,7 +648,8 @@ test ('hydrate every route with stored browser preferences', { }` }, sessionId) const evaluate = async expression => (await send ('Runtime.evaluate', { expression, returnByValue: true }, sessionId)).result.value - for (const route of routes.filter (entry => !entry.handle?.redirectTo)) { + for (const route of routes.filter (entry => !entry.handle?.redirectTo + && (process.env.SSR_COUNTS_ONLY !== '1' || entry.path.startsWith ('counts')))) { for (const locale of ['ja', 'ain-Kana']) { await t.test (`hydrate ${locale} ${route.path}`, async () => { const path = route.path.replace (':identifier', 'home').replace (/:id|:title/g, '1') @@ -608,6 +675,65 @@ test ('hydrate every route with stored browser preferences', { }) } } + for (const width of [375, 1240]) { + await send ('Emulation.setDeviceMetricsOverride', { + width, height: 900, deviceScaleFactor: 1, mobile: width < 768 }, sessionId) + for (const resource of ['posts', 'tags']) { + await t.test (`counts layout ${width} ${resource}`, async () => { + await send ('Page.navigate', { + url: `${httpOrigin}/ja/counts/${resource}` }, sessionId) + let ready = false + for (let attempt = 0; attempt < 100; ++attempt) { + await new Promise (resolve => setTimeout (resolve, 75)) + try { + ready = await evaluate (`document.querySelector('tbody tr')!=null`) + } catch {} + if (ready) break + } + assert.ok (ready, 'history query results must render') + const layout = await evaluate (`(()=>{ + const nav=document.getElementById('counts-targets'); + const main=document.querySelector('main').getBoundingClientRect(); + return {visible:nav.getClientRects().length>0, + right:nav.getBoundingClientRect().right,left:main.left,width:main.width}; + })()`) + assert.ok (width >= 768 ? layout.visible && layout.right <= layout.left + : !layout.visible && layout.width >= width - 16, + 'the sidebar sits beside the table on desktop and collapses on mobile') + const toggle = `Array.from(document.querySelectorAll('button')) + .find(button=>button.textContent.includes('集計条件'))` + assert.equal (await evaluate (`(${toggle}).getAttribute('aria-expanded')`), 'false') + await evaluate (`(${toggle}).click()`) + const positions = await evaluate (`(()=>{ + const output=document.querySelector('[data-counts-output]').getBoundingClientRect(); + const filters=document.querySelector('[data-counts-filters]').getBoundingClientRect(); + return {left:output.left,right:filters.left,top:output.top,bottom:filters.top}; + })()`) + assert.ok (width >= 768 ? positions.left < positions.right + : positions.top < positions.bottom, 'output precedes filters in the layout') + assert.ok (await evaluate ('document.documentElement.scrollWidth<=innerWidth')) + if (resource === 'posts' && process.env.SSR_COUNTS_ONLY === '1') { + const { writeFile } = await import ('node:fs/promises') + const shot = await send ('Page.captureScreenshot', { format: 'png' }, sessionId) + await writeFile (`/tmp/counts-${width}-conditions.png`, + Buffer.from (shot.data, 'base64')) + } + await evaluate (`(${toggle}).click()`) + for (const dark of [false, true]) { + await evaluate (`document.documentElement.classList.toggle('dark',${dark})`) + assert.ok (await evaluate ('document.documentElement.scrollWidth<=innerWidth'), + 'counts controls must fit the viewport in both themes') + if (resource === 'posts' && process.env.SSR_COUNTS_ONLY === '1') { + const { writeFile } = await import ('node:fs/promises') + const shot = await send ('Page.captureScreenshot', { format: 'png' }, sessionId) + await writeFile (`/tmp/counts-${width}-${dark ? 'dark' : 'light'}.png`, + Buffer.from (shot.data, 'base64')) + } + } + }) + } + } + if (process.env.SSR_COUNTS_ONLY === '1') return probeActive = true for (const width of [375, 1240]) { await send ('Emulation.setDeviceMetricsOverride', { diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index d083062..a6799ad 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -1,6 +1,6 @@ import { useQuery, useQueryClient } from '@tanstack/react-query' import { AnimatePresence, LayoutGroup, MotionConfig, motion } from 'framer-motion' -import { createElement, Suspense, lazy, useEffect, +import { createElement, Suspense, lazy, useCallback, useEffect, useRef, useLayoutEffect, useMemo, useState } from 'react' import { useTranslation } from 'react-i18next' import { createBrowserRouter, @@ -22,7 +22,8 @@ import { initialiseUserSession } from '@/lib/users' import { prepareHelpPage } from '@/lib/help' import { fetchLocales, isLocaleCode, legacyLocalePath, switchLocalePath, unlocalisedPath } from '@/lib/locales' -import { localesKeys, sessionKeys } from '@/lib/queryKeys' +import { isAuthorityDependentQuery, localesKeys, sessionKeys } from '@/lib/queryKeys' +import { isCountsPath } from '@/lib/routeLocales' import { switchPostFilterLocale } from '@/lib/posts' import { switchWikiLocale } from '@/lib/wiki' import { useLocale } from '@/lib/useLocale' @@ -63,7 +64,10 @@ const lazyPage = ( }) const Deferred = lazy (preload) // A preloaded page must render synchronously on the first hydration pass. - const Page = (props: Props) => createElement (loaded ?? Deferred, props) + const Page = (props: Props) => { + const [Component] = useState (() => loaded ?? Deferred) + return createElement (Component, props) + } return Object.assign (Page, { preload }) } @@ -77,6 +81,7 @@ const MaterialListPage = lazyPage (() => import ('@/pages/materials/MaterialList const MaterialNewPage = lazyPage (() => import ('@/pages/materials/MaterialNewPage')) const MaterialSyncSuppressionsPage = lazyPage ( () => import ('@/pages/materials/MaterialSyncSuppressionsPage')) +const CountsResourcePage = lazyPage (() => import ('@/pages/counts/CountsResourcePage')) const MorePage = lazyPage (() => import ('@/pages/MorePage')) const GekanatorUnavailablePage = lazyPage (() => import ('@/pages/GekanatorUnavailablePage')) const NicoTagListPage = lazyPage (() => import ('@/pages/tags/NicoTagListPage')) @@ -194,6 +199,22 @@ export const pageRoutes = ( { id: 'gekanator', path: 'gekanator', element: , handle: { locale: 'ja', sitemap: ['static'], preload: locale => locale === 'ja' ? GekanatorPage.preload () : GekanatorUnavailablePage.preload () } }, + { path: 'counts', element: , + handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex', + switchLocale: switchPostFilterLocale } }, + { path: 'counts/posts', element: , + handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex', + switchLocale: switchPostFilterLocale } }, + { path: 'counts/tags', element: , + handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } }, + { path: 'counts/nico-tags', element: , + handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } }, + { path: 'counts/tag-descriptions', element: , + handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } }, + { path: 'counts/tag-groups', element: , + handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } }, + { path: 'counts/materials', element: , + handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } }, { path: 'more', element: , handle: { preload: MorePage.preload, sitemap: ['static'] } }, { path: '*', element: , handle: { notFound: true } }] @@ -206,8 +227,11 @@ const matchPageRoutes = (pathname: string) => export const preloadPageRoute = async ( pathname: string, locale: LocaleCode, + intent: 'navigation' | 'prefetch' = 'navigation', ): Promise => { + if (intent === 'prefetch' && isCountsPath (pathname)) return await Promise.all ((matchPageRoutes (pathname) ?? []) + .filter (match => intent === 'navigation' || match.route.handle?.prefetch !== false) .map (match => match.route.handle?.preload?.(locale))) } @@ -244,15 +268,32 @@ const RoutedApp = ({ locales }: { locales: AvailableLocale[] }) => { const location = useLocation () const locale = useLocale () const route = resolvePageRoute (location.pathname) - const robots = route?.route.handle?.robots + const robots = isCountsPath (location.pathname) ? 'noindex' : route?.route.handle?.robots + const pagePath = unlocalisedPath (location.pathname) + const countsPage = pagePath === '/counts' || pagePath.startsWith ('/counts/') const switchLocale = async (targetLocale: LocaleCode): Promise => { const resolve = route?.route.handle?.switchLocale return (resolve != null && route != null) ? resolve ({ location, params: route.params, locale, targetLocale }) : switchLocalePath (location, targetLocale) } - const [user, setUser] = useState ( + const [user, setUserState] = useState ( () => client.getQueryData (sessionKeys.current) ?? null) + const authority = useRef (user) + const setUser = useCallback>> (action => { + const next = typeof action === 'function' ? action (authority.current) : action + if (next?.id !== authority.current?.id || next?.role !== authority.current?.role) + { + const filters = { + predicate: (query: { queryKey: readonly unknown[] }) => + isAuthorityDependentQuery (query.queryKey) } + void client.cancelQueries (filters) + client.removeQueries (filters) + } + authority.current = next + client.setQueryData (sessionKeys.current, next) + setUserState (next) + }, [client]) const [status, setStatus] = useState (200) const behaviourSettings = useClientBehaviourSettings () const animationMode = behaviourSettings.animation ?? 'normal' @@ -301,7 +342,7 @@ const RoutedApp = ({ locales }: { locales: AvailableLocale[] }) => { } }) () return () => { cancelled = true } - }, []) + }, [setUser]) useEffect (() => { if (!(user)) @@ -350,7 +391,7 @@ const RoutedApp = ({ locales }: { locales: AvailableLocale[] }) => { }> @@ -393,7 +434,7 @@ const LocaleRoute: FC = () => { void i18n.changeLanguage (activeLocale) }, [activeLocale, i18n]) - const redirect = legacyLocalePath (location) + const redirect = legacyLocalePath (location, pathname => resolvePageRoute (pathname) != null) if (!(isLocaleCode (locale)) && redirect == null) return if (isError && data == null) diff --git a/frontend/src/components/PrefetchLink.tsx b/frontend/src/components/PrefetchLink.tsx index 01a4c01..27aefc7 100644 --- a/frontend/src/components/PrefetchLink.tsx +++ b/frontend/src/components/PrefetchLink.tsx @@ -7,6 +7,7 @@ import { localiseTo } from '@/lib/locales' import { useLocale, useLocaleNavigate } from '@/lib/useLocale' import { useOverlayStore } from '@/components/RouteBlockerOverlay' import { prefetchForURL } from '@/lib/prefetchers' +import { isCountsPath } from '@/lib/routeLocales' import { useClientBehaviourSettings } from '@/lib/useClientBehaviourSettings' import { cn } from '@/lib/utils' import { currentRenderContext } from '@/lib/renderContext' @@ -53,6 +54,7 @@ export default forwardRef (({ const setOverlay = useOverlayStore (s => s.setActive) const doPrefetch = async () => { + if (isCountsPath (new URL (url).pathname)) return true try { await prefetchForURL (qc, url) @@ -94,6 +96,12 @@ export default forwardRef (({ ev.preventDefault () + if (isCountsPath (new URL (url).pathname)) + { + navigate (path, { replace, ...(state && { state }) }) + return + } + flushSync (() => { setOverlay (true) }) diff --git a/frontend/src/components/TagSidebar.tsx b/frontend/src/components/TagSidebar.tsx index 7e8550a..26f21ab 100644 --- a/frontend/src/components/TagSidebar.tsx +++ b/frontend/src/components/TagSidebar.tsx @@ -4,6 +4,7 @@ import { useLocation } from 'react-router-dom' import { useLocale, useLocaleNavigate } from '@/lib/useLocale' import { useUITranslation } from '@/lib/useUITranslation' +import PrefetchLink from '@/components/PrefetchLink' import TagLink from '@/components/TagLink' import TagSearch from '@/components/TagSearch' import SectionTitle from '@/components/common/SectionTitle' @@ -42,6 +43,12 @@ const TagSidebar: FC = ({ posts, onClick }) => { const query = new URLSearchParams (location.search) const tagsQuery = query.get ('tags') ?? '' const anyFlg = query.get ('match') === 'any' + const countsQuery = new URLSearchParams ({ tags: tagsQuery }) + for (const [key, value] of query) + { + if (key === 'match') + countsQuery.append (key, value) + } useEffect (() => { const tagsTmp: TagByCategory = { } @@ -93,25 +100,37 @@ const TagSidebar: FC = ({ posts, onClick }) => { ))) : [])} {label ('labels.relations')} - { - ev.preventDefault () - void ((async () => { - try - { - const data = await apiGet ('/posts/random', - { locale, params: { tags: tagsQuery.split (' ').filter (e => e !== '').join (' '), - match: (anyFlg ? 'any' : 'all') } }) - navigate (`/posts/${ data.id }`) - } - catch - { - ; - } - }) ()) - }}> - {label ('actions.random')} - + ) return ( diff --git a/frontend/src/components/TopNav.tsx b/frontend/src/components/TopNav.tsx index 4904b96..22dc2dd 100644 --- a/frontend/src/components/TopNav.tsx +++ b/frontend/src/components/TopNav.tsx @@ -120,6 +120,11 @@ export const menuOutline = ( { ...message ('entertainment'), visible: false, subMenu: [ { ...message ('theatre'), to: '/theatres/1' }, { ...message ('gekanator'), to: '/gekanator' }] }, + { ...message ('counts'), + name: `${ message ('counts').name } (β)`, visible: false, subMenu: [ + { ...message ('posts'), to: '/counts/posts' }, + { ...message ('tags'), to: '/counts/tags' }, + { ...message ('externalTags'), to: '/counts/nico-tags' }] }, { ...message ('users'), to: '/users/settings', visible: false, subMenu: [ { ...message ('list'), to: '/users', visible: false }, { ...message ('currentUser'), to: `/users/${ user?.id }`, visible: false }, diff --git a/frontend/src/components/common/AnimatedCollapse.tsx b/frontend/src/components/common/AnimatedCollapse.tsx new file mode 100644 index 0000000..81de06b --- /dev/null +++ b/frontend/src/components/common/AnimatedCollapse.tsx @@ -0,0 +1,22 @@ +import { motion } from 'framer-motion' + +import { useClientMotion } from '@/lib/useClientMotion' +import { cn } from '@/lib/utils' + +import type { ReactNode } from 'react' + + +const AnimatedCollapse = ({ open, children, id, className }: { + open: boolean; children: ReactNode; id?: string; className?: string }, +) => { + const motionSettings = useClientMotion () + return + {children} + +} + + +export default AnimatedCollapse diff --git a/frontend/src/components/common/DateTimeField.tsx b/frontend/src/components/common/DateTimeField.tsx index 74fbec8..2ea074e 100644 --- a/frontend/src/components/common/DateTimeField.tsx +++ b/frontend/src/components/common/DateTimeField.tsx @@ -8,13 +8,13 @@ import type { ComponentPropsWithoutRef, FC, FocusEvent } from 'react' const pad = (n: number): string => n.toString ().padStart (2, '0') -const toDateTimeLocalValue = (value: Date) => { +const toDateTimeLocalValue = (value: Date, seconds = false) => { const y = value.getFullYear () const m = pad (value.getMonth () + 1) const day = pad (value.getDate ()) const h = pad (value.getHours ()) const min = pad (value.getMinutes ()) - return `${ y }-${ m }-${ day }T${ h }:${ min }` + return `${ y }-${ m }-${ day }T${ h }:${ min }${seconds ? `:${pad (value.getSeconds ())}` : ''}` } @@ -37,15 +37,17 @@ type Props = Omit, 'onChange'> & { onChange?: (isoUTC: string | null) => void className?: string onBlur?: (ev: FocusEvent) => void - invalid?: boolean } + invalid?: boolean + precision?: 'minute' | 'second' } -const DateTimeField: FC = ({ value, onChange, className, onBlur, invalid, ...rest }) => { +const DateTimeField: FC = props => { + const { value, onChange, className, onBlur, invalid, precision = 'minute', ...rest } = props const [local, setLocal] = useState ('') useEffect (() => { - setLocal (value ? toDateTimeLocalValue (new Date (value)) : '') - }, [value]) + setLocal (value ? toDateTimeLocalValue (new Date (value), precision === 'second') : '') + }, [value, precision]) return ( = ({ value, onChange, className, onBlur, invalid, 'focus:ring-2 focus:ring-blue-200']), className)} type="datetime-local" - step={60} + step={precision === 'second' ? 1 : 60} value={local} aria-invalid={invalid} onChange={ev => { const v = ev.target.value setLocal (v) - onChange?.(v ? toMinutePrecisionIsoUtc (v) : null) + const iso = v ? (precision === 'second' + ? new Date (v).toISOString () : toMinutePrecisionIsoUtc (v)) : null + onChange?.(iso) }} onBlur={onBlur}/>) } diff --git a/frontend/src/components/common/HelpPopover.tsx b/frontend/src/components/common/HelpPopover.tsx new file mode 100644 index 0000000..3ae8bbd --- /dev/null +++ b/frontend/src/components/common/HelpPopover.tsx @@ -0,0 +1,35 @@ +import { useId, useRef } from 'react' +import { CircleHelp } from 'lucide-react' + +import type { ReactNode } from 'react' + + +const HelpPopover = ({ label, children }: { + label: { text: string; language: string }; children: ReactNode }, +) => { + const id = useId () + const trigger = useRef (null) + const panel = useRef (null) + return <> + + + +} + + +export default HelpPopover diff --git a/frontend/src/components/common/SortableColumns.tsx b/frontend/src/components/common/SortableColumns.tsx new file mode 100644 index 0000000..e8c5306 --- /dev/null +++ b/frontend/src/components/common/SortableColumns.tsx @@ -0,0 +1,77 @@ +import { DndContext, DragOverlay, KeyboardSensor, MouseSensor, TouchSensor, + closestCenter, useSensor, useSensors } from '@dnd-kit/core' +import { SortableContext, arrayMove, horizontalListSortingStrategy, + sortableKeyboardCoordinates } from '@dnd-kit/sortable' +import { useEffect, useId, useRef, useState } from 'react' +import { createPortal } from 'react-dom' +import { CSS } from '@dnd-kit/utilities' + +import { useClientMotion } from '@/lib/useClientMotion' + +import type { ReactNode } from 'react' + + +const SortableColumns = ({ ids, fixed = [], name, onChange, children }: { + ids: string[]; fixed?: string[]; name: (id: string) => { text: string; language: string } + onChange: (ids: string[]) => void; children: (ids: string[]) => ReactNode }, +) => { + const [mounted, setMounted] = useState (false) + const scope = useId () + useEffect (() => setMounted (true), []) + const [active, setActive] = useState (null) + const drag = useRef<{ id: string; ids: string[] } | null> (null) + const motion = useClientMotion () + const sensors = useSensors ( + useSensor (MouseSensor, { activationConstraint: { distance: 4 } }), + useSensor (TouchSensor, { activationConstraint: { delay: 350, tolerance: 8 } }), + useSensor (KeyboardSensor, { coordinateGetter: sortableKeyboardCoordinates })) + const move = (current: string[], from: string, to: string) => { + if (fixed.includes (from) || fixed.includes (to)) return current + const source = current.indexOf (from) + const target = current.indexOf (to) + return source < 0 || target < 0 || source === target + ? current : arrayMove (current, source, target) + } + const reset = () => { + drag.current = null + setActive (null) + } + return name (String (event.active.id)).text, + onDragOver: event => event.over == null ? undefined : name (String (event.over.id)).text, + onDragMove: () => undefined, onDragEnd: () => undefined, + onDragCancel: () => undefined }, + screenReaderInstructions: { draggable: active == null ? '' : name (active).text } }} + onDragStart={event => { + const id = String (event.active.id) + drag.current = { id, ids } + setActive (id) + }} onDragCancel={reset} onDragEnd={event => { + const session = drag.current + reset () + if (session == null || event.over == null || session.id !== String (event.active.id) + || session.ids.length !== ids.length + || session.ids.some ((id, index) => id !== ids[index])) return + const next = move (ids, session.id, String (event.over.id)) + if (next.some ((id, index) => id !== ids[index])) onChange (next) + }}> + + {children (ids)} + + {mounted && typeof document !== 'undefined' && createPortal ( [ + { transform: CSS.Translate.toString (transform.initial) }, + { transform: CSS.Translate.toString (transform.final) }] }}> + {active != null &&
{name (active).text}
} + , document.body)} + +} + +export default SortableColumns diff --git a/frontend/src/components/common/SortableItem.tsx b/frontend/src/components/common/SortableItem.tsx new file mode 100644 index 0000000..a72abf7 --- /dev/null +++ b/frontend/src/components/common/SortableItem.tsx @@ -0,0 +1,60 @@ +import { defaultAnimateLayoutChanges, useSortable } from '@dnd-kit/sortable' +import { CSS } from '@dnd-kit/utilities' + +import { useClientMotion } from '@/lib/useClientMotion' +import { cn } from '@/lib/utils' + +import type { CSSProperties, ReactNode } from 'react' + +type Name = { text: string; language: string } + + +const SortableItem = ({ id, name, fixed, children, moveGroup, handleId, renderShell }: { + id: string; name: Name; fixed?: boolean; children: ReactNode + moveGroup: (direction: number) => void; handleId: string + renderShell?: (shell: { ref: (node: HTMLElement | null) => void + style: CSSProperties; handle: ReactNode; dragging: boolean }) => ReactNode }, +) => { + const motion = useClientMotion () + const item = useSortable ({ id, disabled: fixed, + transition: { duration: motion.duration ('reorder'), easing: motion.easing }, + animateLayoutChanges: options => !motion.reduced + && defaultAnimateLayoutChanges ({ ...options, wasDragging: true }) }) + const handle = fixed ? : + const style = { transform: CSS.Translate.toString (item.transform), transition: item.transition } + if (renderShell != null) + return renderShell ({ ref: item.setNodeRef, style, handle, dragging: item.isDragging }) + return
+
+ {handle} +
{children}
+
+
+} + + +export default SortableItem diff --git a/frontend/src/components/common/SortableList.tsx b/frontend/src/components/common/SortableList.tsx new file mode 100644 index 0000000..73025f7 --- /dev/null +++ b/frontend/src/components/common/SortableList.tsx @@ -0,0 +1,163 @@ +import { DndContext, DragOverlay, KeyboardSensor, MouseSensor, TouchSensor, + closestCenter, pointerWithin, useDroppable, useSensor, useSensors } + from '@dnd-kit/core' +import { SortableContext, arrayMove, rectSortingStrategy, + sortableKeyboardCoordinates } from '@dnd-kit/sortable' +import { useEffect, useId, useRef, useState } from 'react' +import { createPortal } from 'react-dom' +import { CSS } from '@dnd-kit/utilities' + +import SortableItem from '@/components/common/SortableItem' +import { useClientMotion } from '@/lib/useClientMotion' + +import type { ReactNode } from 'react' + +type Groups = Record +type Name = { text: string; language: string } + + +const sameGroups = (left: Groups, right: Groups): boolean => + Object.keys (left).length === Object.keys (right).length + && ( + Object.keys (left).every (key => ( + left[key].length === right[key]?.length + && left[key].every ((id, index) => id === right[key][index])))) + + +const Group = ({ id, children }: { id: string; children: ReactNode }) => { + const drop = useDroppable ({ id: `group:${id}` }) + return
{children}
+} + + +const SortableList = ( + { groups, name, handleName, renderItem, renderGroup, onChange, fixed = [] }: { + groups: Groups; name: (id: string) => Name + handleName?: (id: string) => Name + renderItem: (id: string, group: string) => ReactNode + renderGroup: (id: string, content: ReactNode, over: boolean) => ReactNode + onChange: (groups: Groups) => void; fixed?: string[] }, +) => { + const [mounted, setMounted] = useState (false) + useEffect (() => setMounted (true), []) + const drag = useRef<{ id: string; groups: Groups } | null> (null) + const scope = useId () + const [active, setActive] = useState (null) + const [overGroup, setOverGroup] = useState (null) + const motion = useClientMotion () + const accessibleName = handleName ?? name + const sensors = useSensors ( + useSensor (MouseSensor, { activationConstraint: { distance: 4 } }), + useSensor (TouchSensor, { activationConstraint: { delay: 350, tolerance: 8 } }), + useSensor (KeyboardSensor, { coordinateGetter: sortableKeyboardCoordinates })) + const owner = (id: string, value: Groups) => Object.keys (value) + .find (group => value[group].includes (id)) + const relocate = (value: Groups, id: string, over: string): Groups => { + const source = owner (id, value) + const target = over.startsWith ('group:') ? over.slice (6) : owner (over, value) + if (source == null || target == null || value[target] == null + || fixed.includes (id) || fixed.includes (over)) + return value + if (source === target) + { + const from = value[source].indexOf (id) + const to = over.startsWith ('group:') + ? value[target].length - 1 : value[target].indexOf (over) + if (from === to || to < 0) return value + return { ...value, [source]: arrayMove (value[source], from, to) } + } + const destination = value[target].filter (item => item !== id) + const position = destination.indexOf (over) + destination.splice (position < 0 ? destination.length : position, 0, id) + return { ...value, [source]: value[source].filter (item => item !== id), + [target]: destination } + } + const moveGroup = (id: string, direction: number) => { + if (drag.current != null) return + const keys = Object.keys (groups) + const group = owner (id, groups) + const target = keys[keys.indexOf (group ?? '') + direction] + if (target != null) + { + const next = relocate (groups, id, `group:${target}`) + if (sameGroups (groups, next)) return + onChange (next) + requestAnimationFrame (() => document.getElementById (`${scope}-${id}`)?.focus ()) + } + } + const reset = () => { + drag.current = null + setActive (null) + setOverGroup (null) + } + return ( + { + const hits = pointerWithin (args) + const group = hits.find (hit => String (hit.id).startsWith ('group:')) + const groupName = group?.id.toString ().slice (6) + const item = groupName != null && hits.some (hit => + owner (String (hit.id), groups) === groupName) + return group != null && !item ? [group] : closestCenter (args) + }} + accessibility={{ announcements: { + onDragStart: event => accessibleName (String (event.active.id)).text, + onDragOver: event => event.over == null ? undefined : ( + event.over.id.toString ().startsWith ('group:') ? undefined + : accessibleName (String (event.over.id)).text), + onDragMove: () => undefined, onDragEnd: () => undefined, + onDragCancel: () => undefined }, + screenReaderInstructions: { + draggable: active == null ? '' : accessibleName (active).text } }} + onDragStart={event => { + const id = String (event.active.id) + drag.current = { id, groups } + setActive (id) + setOverGroup (owner (id, groups) ?? null) + }} + onDragOver={event => { + const id = event.over?.id.toString () + const group = id == null ? null + : (id.startsWith ('group:') ? id.slice (6) : owner (id, groups) ?? null) + setOverGroup (previous => previous === group ? previous : group) + }} onDragCancel={reset} onDragEnd={event => { + const session = drag.current + reset () + if (session == null || event.over == null || session.id !== String (event.active.id) + || !sameGroups (session.groups, groups)) return + const next = relocate (groups, session.id, String (event.over.id)) + if (!sameGroups (groups, next)) onChange (next) + }}> + {/* Keep indexes and droppable geometry stable until the drop is committed. */} + {Object.entries (groups).map (([group, ids]) => + {renderGroup (group, +
+ {ids.map (id => moveGroup (id, direction)}> + {renderItem (id, group)} + )} +
+ , overGroup === group)} + )} + {mounted && typeof document !== 'undefined' && createPortal ( + [ + { transform: CSS.Translate.toString (transform.initial) }, + { transform: CSS.Translate.toString (transform.final) }] }}> + {active == null ? null :
+ {name (active).text}
} + , document.body)} + ) +} + + +export default SortableList diff --git a/frontend/src/components/common/TagInput.tsx b/frontend/src/components/common/TagInput.tsx index be12274..d2967ff 100644 --- a/frontend/src/components/common/TagInput.tsx +++ b/frontend/src/components/common/TagInput.tsx @@ -12,6 +12,7 @@ import type { Tag } from '@/types' type Props = { includeNico?: boolean + includeUnused?: boolean describedBy?: string invalid?: boolean value: string @@ -20,6 +21,7 @@ type Props = { const TagInput: FC = ({ includeNico = false, + includeUnused = false, describedBy, invalid, value, @@ -57,11 +59,11 @@ const TagInput: FC = ({ { const tags = await apiGet ('/tags/autocomplete', { locale, signal: controller.signal, - params: { q: request.query, nico: includeNico ? '1' : '0' } }) + params: { q: request.query.replace (/^not:/i, ''), nico: includeNico ? '1' : '0' } }) if (active) { setResult ({ locale, includeNico, request, - tags: tags.filter (tag => tag.postCount > 0) }) + tags: includeUnused ? tags : tags.filter (tag => tag.postCount > 0) }) } } catch @@ -74,7 +76,7 @@ const TagInput: FC = ({ active = false controller.abort () } - }, [request, locale, includeNico]) + }, [request, locale, includeNico, includeUnused]) useEffect (() => { setActiveIndex (-1) @@ -85,7 +87,8 @@ const TagInput: FC = ({ // TODO: TagSearch からのコピペのため，共通化を考へる． const handleTagSelect = (tag: Tag) => { const parts = value?.split (' ') - parts[parts.length - 1] = tag.name + const negative = /^not:/i.test (parts[parts.length - 1]) + parts[parts.length - 1] = `${negative ? 'not:' : ''}${tag.name}` setValue (parts.join (' ') + ' ') setRequest ({ query: '' }) setActiveIndex (-1) diff --git a/frontend/src/components/counts/CountsAdditionalConditions.tsx b/frontend/src/components/counts/CountsAdditionalConditions.tsx new file mode 100644 index 0000000..aa170cf --- /dev/null +++ b/frontend/src/components/counts/CountsAdditionalConditions.tsx @@ -0,0 +1,213 @@ +import { useEffect, useState } from 'react' + +import CountsConditionValues from '@/components/counts/CountsConditionValues' +import DateTimeField from '@/components/common/DateTimeField' +import { Button } from '@/components/ui/button' +import { countsConditionInputs, countsEventTypes, countsFieldLabel, countsFilterFields, + countsSetOperators, useCountsDefinition } + from '@/lib/countsDefinitions' +import useDialogue from '@/lib/dialogues/useDialogue' +import { countsConditionValue } from '@/lib/countsDisplay' +import { countsAxisLabel } from '@/lib/countsDefinitions' +import { useLocale } from '@/lib/useLocale' +import { inputClass } from '@/lib/utils' +import { useUITranslation } from '@/lib/useUITranslation' + +import { readCountState, encodeCountState } from '@/lib/queryState' + +import type { CountsResult } from '@/lib/counts' +import type { DialogueFormControls } from '@/lib/dialogues/useDialogue' +import type { CountsCondition, CountsDefinition } from '@/lib/countsDefinitions' + + +const ConditionEditor = ( + { definition, initial, controls, save }: { + definition: CountsDefinition; initial?: CountsCondition + controls: DialogueFormControls + save: (condition: CountsCondition) => void }, +) => { + const { label, message } = useUITranslation ('counts') + const fields = definition.filterFields ?? definition.fields + const fieldName = (key: string) => (fields.find (item => item.key === key)?.labels + ?? [countsFieldLabel (key)]).map (label => message ( + label, fields.find (item => item.key === key)?.labelValues).text).join (' › ') + const [condition, setCondition] = useState (() => { + const next = { ...initial, field: initial?.field ?? 'resource_id', + operator: initial?.operator ?? 'eq' } + if (next.operator !== 'range') delete next.end + return next + }) + const { field, operator } = condition + const operators = fields.find (item => item.key === field)?.operators ?? [] + const inputs = countsConditionInputs (fields.find (item => item.key === field), operator) + useEffect (() => { + controls.setActions ([{ label: label (initial == null ? 'add' : 'shared:actions.edit'), + onSelect: () => save (condition) }]) + }, [controls, condition, initial, label, save]) + return
+ + + {inputs.length > 1 && } +
+} + + +const CountsAdditionalConditions = ({ params, update, conditionLabels }: { + conditionLabels?: CountsResult['conditionLabels'] + params: URLSearchParams + update: (key: string, value: string) => void }, +) => { + const { label, message } = useUITranslation ('counts') + const definition = useCountsDefinition () + const dialogue = useDialogue () + const locale = useLocale () + const conditions = readCountState (params, 'conditions', []) + const metadata = definition.filterFields ?? definition.fields + const fields = metadata.filter (field => field.operators.length > 0 && !field.legacy) + const fieldName = (key: string) => (metadata.find (item => item.key === key)?.labels + ?? [countsFieldLabel (key)]).map (label => message ( + label, metadata.find (item => item.key === key)?.labelValues).text).join (' › ') + const write = (next: CountsCondition[]) => update ( + 'conditions', encodeCountState ('conditions', next)) + const replace = (index: number, condition: CountsCondition) => write ( + conditions.map ((item, position) => position === index ? condition : item)) + const patch = (index: number, change: Partial) => + replace (index, { ...conditions[index], ...change }) + const edit = (index?: number) => { + const initial = index == null ? undefined : conditions[index] + const save = (condition: CountsCondition) => { + if (index == null) write ([...conditions, condition]) + else replace (index, condition) + } + void dialogue.form ({ title: label ('additionalConditions'), footerLayout: 'inline', + body: controls => }) + } + const valueInput = (condition: CountsCondition, index: number, key: 'value' | 'end') => { + const type = countsFilterFields (definition)[condition.field] + const aria = message (key === 'end' ? 'through' + : (condition.operator === 'range' ? 'from' : 'value')) + if (type === 'boolean' || condition.field === 'event_type') + { + const options: string[] = type === 'boolean' ? ['true', 'false'] : [...countsEventTypes] + const namespace = type === 'boolean' ? 'truthValues' : 'events' + return + } + if (type === 'datetime') + return patch (index, { [key]: value ?? '' })}/> + return patch (index, { [key]: event.target.value })}/> + } + return
+ {label ('additionalConditions')} + + {conditions.map ((condition, index) => condition.operator === 'axis_eq' ? ( +
+ {countsAxisLabel (condition, definition, locale).text} + {label ('operatorSymbols.eq')} + {countsConditionValue (condition.value, condition, locale, + params.get ('timezone') ?? 'UTC', conditionLabels)} + +
) : ( +
+ + {countsSetOperators.includes (condition.operator) ? ( + field.key === condition.field)!} + definition={definition} params={params} + maximum={Math.max (0, 100 - conditions.reduce ((sum, item, position) => + sum + (position === index ? 0 : item.values?.length ?? 0), 0))} + change={values => patch (index, { values })}/>) : ( +
+ {valueInput (condition, index, 'value')} + {condition.operator === 'range' && <> + + {valueInput (condition, index, 'end')} + } +
)} +
+ + +
+
))} +
+} + + +export default CountsAdditionalConditions diff --git a/frontend/src/components/counts/CountsAxes.tsx b/frontend/src/components/counts/CountsAxes.tsx new file mode 100644 index 0000000..d878d86 --- /dev/null +++ b/frontend/src/components/counts/CountsAxes.tsx @@ -0,0 +1,99 @@ +import LocalisedText from '@/components/LocalisedText' +import SortableList from '@/components/common/SortableList' +import { countsAxisFields, countsFilterFields, countsFieldLabel, + useCountsDefinition } + from '@/lib/countsDefinitions' +import { useClientMotion } from '@/lib/useClientMotion' +import { cn, inputClass } from '@/lib/utils' +import { useUITranslation } from '@/lib/useUITranslation' + +import { readCountState, encodeCountState } from '@/lib/queryState' + +import type { CountsAxis } from '@/lib/countsDefinitions' + + +const CountsAxes = ({ params, update }: { + params: URLSearchParams; update: (key: string, value: string) => void }, +) => { + const { label, message } = useUITranslation ('counts') + const definition = useCountsDefinition () + const motion = useClientMotion () + const available = countsAxisFields (definition, params.get ('definition') === 'history') + const rows = readCountState (params, 'rows', []) + const cols = readCountState (params, 'cols', []) + const axes = [...rows, ...cols] + const groups = { + available: available.map (field => field.key).filter (id => !axes.some ( + axis => axis.field === id)), + rows: rows.map (axis => axis.field), cols: cols.map (axis => axis.field) } + const name = (id: string) => { + const axis = axes.find (item => item.field === id) + return axis == null ? message (available.find (field => field.key === id)?.label ?? 'axis') + : message (countsFieldLabel (axis.field)) + } + const commit = (next: Record) => { + for (const group of ['rows', 'cols'] as const) + { + if (next[group].join (' ') === groups[group].join (' ')) continue + const selected = next[group].map (field => axes.find (axis => axis.field === field) + ?? { field, ...(countsFilterFields (definition)[field] === 'datetime' + ? { grain: 'month' } : { }) }) + update (group, encodeCountState (group, selected)) + } + } + const place = (id: string, target?: 'rows' | 'cols') => { + const next = Object.fromEntries (Object.entries (groups).map (([key, ids]) => [ + key, ids.filter (value => value !== id)])) + if (target != null) next[target].push (id) + else next.available.push (id) + commit (next) + } + return
+ + {label (id === 'available' ? 'axisItems' : id)} +
{content}
+
} renderItem={(id, group) => { + const axis = axes.find (axis => axis.field === id) + return
+ + {name (id).text} + + {group === 'available' && + {(['rows', 'cols'] as const).map (target => )} + } + {group !== 'available' && <> + {countsFilterFields (definition)[id] === 'datetime' && } + + } +
+ }}/> +} + + +export default CountsAxes diff --git a/frontend/src/components/counts/CountsCandidatePicker.tsx b/frontend/src/components/counts/CountsCandidatePicker.tsx new file mode 100644 index 0000000..ad3ba60 --- /dev/null +++ b/frontend/src/components/counts/CountsCandidatePicker.tsx @@ -0,0 +1,139 @@ +import { useQuery } from '@tanstack/react-query' +import { useEffect, useRef, useState } from 'react' + +import { Button } from '@/components/ui/button' +import { countsUseLatest } from '@/lib/counts' +import { fetchCountsCandidates } from '@/lib/countsDefinitions' +import { countsKeys } from '@/lib/queryKeys' +import { useLocale } from '@/lib/useLocale' +import { useUITranslation } from '@/lib/useUITranslation' +import { inputClass } from '@/lib/utils' + +import type { DialogueFormControls } from '@/lib/dialogues/useDialogue' +import type { CountsCandidate, CountsDefinition, CountsFieldMetadata } + from '@/lib/countsDefinitions' + + +const CountsCandidatePicker = ( + { field, definition, params, initial, maximum, controls, save }: { + field: CountsFieldMetadata; params: URLSearchParams + definition: CountsDefinition + initial: (string | number | boolean)[]; maximum: number + controls: DialogueFormControls; save: (values: string[]) => void }, +) => { + const { label, message } = useUITranslation ('counts') + const locale = useLocale () + const [selected, setSelected] = useState (() => new Set (initial.map (String))) + const [columns, setColumns] = useState (['id', 'name', 'category', 'kind']) + const [draft, setDraft] = useState ('') + const [search, setSearch] = useState ('') + const [category, setCategory] = useState ('') + const [page, setPage] = useState (1) + const point = useRef (params.get ('execution_at')) + const request = new URLSearchParams ({ field: field.key, query: search, page: String (page) }) + for (const key of ['timezone', 'at', 'execution_at']) + if (params.has (key)) request.set (key, params.get (key)!) + if (countsUseLatest (params)) request.delete ('at') + if (category) request.set ('category', category) + const dynamic = field.candidateSource === 'tags' + const { data, isFetching: fetching, isError: errorFlg } = useQuery ({ + queryKey: countsKeys.candidates (request.toString (), locale), enabled: dynamic, + queryFn: ({ signal }) => { + const pinned = new URLSearchParams (request) + if (point.current != null) pinned.set ('execution_at', point.current) + return fetchCountsCandidates (pinned, locale, signal) + }, retry: false }) + useEffect (() => { + if (data != null) point.current = data.executionAt + }, [data]) + useEffect (() => { + controls.setActions ([{ label: label ('counts:pattern.selection'), + onSelect: () => save ([...selected]) }]) + }, [controls, label, save, selected]) + const fixed: CountsCandidate[] = (field.choices ?? []).map (choice => ({ + value: choice.value, name: message (choice.label).text })) + const rows = dynamic ? data?.rows ?? [] : fixed.filter (row => + row.name?.includes (search)) + const headings: Record = { + id: 'columns.resourceId', name: 'columns.name', category: 'common:categoryLabel', + kind: 'columns.kind' } + const available = dynamic ? ['id', 'name', 'category', 'kind'] : ['name'] + const visible = available.filter (column => columns.includes (column)) + const categories = definition.filterFields?.find (field => + field.candidateSource === 'categories')?.choices ?? [] + const cell = (row: CountsCandidate, column: string) => { + if (column === 'category') return row.category == null ? '' + : message (`common:categories.${row.category}`).text + if (column === 'kind') return row.kind == null ? '' : message (`kinds.${row.kind}`).text + return column === 'id' ? row.id : row.name + } + return
+
+ setDraft (event.target.value)} + onKeyDown={event => { + if (event.key !== 'Enter') return + event.preventDefault () + setSearch (draft) + setPage (1) + }}/> + +
+ {dynamic && } +
+ {label ('counts:pattern.columns')} + {available.map (column => )} +
+

{label ('counts:pattern.value')}{'：'} + {label (dynamic ? 'columns.resourceId' : 'common:categoryLabel')} +

+ + {errorFlg &&

{label ('common:fetchError')}

} +
+ + + {visible.map (column => )} + + {rows.map (row => + + {visible.map (column => )} + )} +
{label ('counts:pattern.selection')} + {label (headings[column])}
= maximum} + onChange={event => setSelected (previous => { + const next = new Set (previous) + if (event.target.checked) next.add (row.value) + else next.delete (row.value) + return next + })}/> + {cell (row, column)}
+
+ {dynamic &&
+ + {page} + +
} +
+} + + +export default CountsCandidatePicker diff --git a/frontend/src/components/counts/CountsCellMenu.tsx b/frontend/src/components/counts/CountsCellMenu.tsx new file mode 100644 index 0000000..9eb51b6 --- /dev/null +++ b/frontend/src/components/counts/CountsCellMenu.tsx @@ -0,0 +1,88 @@ +import { useEffect, useLayoutEffect, useRef } from 'react' +import { createPortal } from 'react-dom' + +import { useUITranslation } from '@/lib/useUITranslation' + +export type CountsMenuAction = 'copy' | 'copyHeaders' | 'row' | 'column' | 'all' + + +const CountsCellMenu = ( + { x, y, act, close }: { + x: number + y: number + act: (action: CountsMenuAction) => void + close: () => void }, +) => { + const root = useRef (null) + const { label, message } = useUITranslation ('counts') + useLayoutEffect (() => { + const menu = root.current + if (menu == null) return + const rect = menu.getBoundingClientRect () + menu.style.left = `${Math.max (4, Math.min (x, window.innerWidth - rect.width - 4))}px` + menu.style.top = `${Math.max (4, Math.min (y, window.innerHeight - rect.height - 4))}px` + menu.querySelector ('button')?.focus ({ preventScroll: true }) + }, [x, y]) + useEffect (() => { + const outside = (event: Event) => { + if (!root.current?.contains (event.target as Node)) close () + } + const escape = (event: KeyboardEvent) => { + if (event.key === 'Escape') + { + event.preventDefault () + event.stopPropagation () + close () + } + } + document.addEventListener ('pointerdown', outside, true) + document.addEventListener ('keydown', escape, true) + window.addEventListener ('resize', close) + document.addEventListener ('scroll', outside, true) + return () => { + document.removeEventListener ('pointerdown', outside, true) + document.removeEventListener ('keydown', escape, true) + window.removeEventListener ('resize', close) + document.removeEventListener ('scroll', outside, true) + } + }, [close]) + const actions: CountsMenuAction[] = ['copy', 'copyHeaders', 'row', 'column', 'all'] + return createPortal ( +
event.preventDefault ()} + onPointerDown={event => event.stopPropagation ()} + onKeyDown={event => { + event.stopPropagation () + if ((event.ctrlKey || event.metaKey) && event.key.toLowerCase () === 'c') + { + event.preventDefault () + act ('copy') + return + } + const buttons = Array.from (root.current?.querySelectorAll ('button') ?? []) + const current = buttons.indexOf (document.activeElement as HTMLButtonElement) + if (['ArrowDown', 'ArrowUp', 'Home', 'End'].includes (event.key)) + { + event.preventDefault () + const next = event.key === 'Home' ? 0 : (event.key === 'End' ? buttons.length - 1 + : (current + (event.key === 'ArrowDown' ? 1 : -1) + buttons.length) + % buttons.length) + buttons[next]?.focus () + } + if (event.key === 'Tab') close () + }}> + {actions.map (action => ( + ))} +
, document.body) +} + + +export default CountsCellMenu diff --git a/frontend/src/components/counts/CountsConditionValues.tsx b/frontend/src/components/counts/CountsConditionValues.tsx new file mode 100644 index 0000000..25b79ec --- /dev/null +++ b/frontend/src/components/counts/CountsConditionValues.tsx @@ -0,0 +1,84 @@ +import { useState } from 'react' + +import CountsCandidatePicker from '@/components/counts/CountsCandidatePicker' +import DateTimeField from '@/components/common/DateTimeField' +import { Button } from '@/components/ui/button' +import useDialogue from '@/lib/dialogues/useDialogue' +import { inputClass } from '@/lib/utils' +import { useUITranslation } from '@/lib/useUITranslation' + +import type { CountsCondition, CountsDefinition, CountsFieldMetadata } + from '@/lib/countsDefinitions' + + +const CountsConditionValues = ( + { condition, field, definition, params, maximum, change }: { + condition: CountsCondition; field: CountsFieldMetadata; definition: CountsDefinition + params: URLSearchParams; maximum: number + change: (values: (string | number | boolean)[]) => void }, +) => { + const { label, message } = useUITranslation ('counts') + const dialogue = useDialogue () + const [draft, setDraft] = useState ('') + const values = condition.values ?? [] + const pick = () => { + void dialogue.form ({ title: label (field.candidateSource === 'tags' + ? 'counts:pattern.link' : 'counts:pattern.pattern'), size: 'large', footerLayout: 'inline', + body: controls => }) + } + if (condition.input === 'list' || field.candidateSource === 'tags') + return + const append = () => { + if (draft === '' || values.length >= maximum + || values.some (value => String (value) === draft)) + return + change ([...values, draft]) + setDraft ('') + } + const input = () => { + if (field.type === 'datetime') + return setDraft (value ?? '')}/> + if (field.type === 'boolean' || field.choices != null) + { + const choices = field.choices ?? ['true', 'false'].map (value => ({ + value, label: `counts:truthValues.${value}` })) + return + } + return setDraft (event.target.value)}/> + } + const display = (value: string | number | boolean) => { + const choice = field.choices?.find (choice => choice.value === String (value)) + if (choice != null) return message (choice.label).text + if (field.type === 'boolean') return message (`truthValues.${value}`).text + return String (value) + } + return
+ {values.map ((value, index) => + {display (value)} + )} + {input ()} + + +
+} + + +export default CountsConditionValues diff --git a/frontend/src/components/counts/CountsControls.test.tsx b/frontend/src/components/counts/CountsControls.test.tsx new file mode 100644 index 0000000..f2b525a --- /dev/null +++ b/frontend/src/components/counts/CountsControls.test.tsx @@ -0,0 +1,128 @@ +import { fireEvent, screen } from '@testing-library/react' +import { useState } from 'react' +import { describe, expect, it, vi } from 'vitest' + +import CountsControls from '@/components/counts/CountsControls' +import CountsFieldTree from '@/components/counts/CountsFieldTree' +import { CountsDefinitionContext } from '@/lib/countsDefinitions' +import { normaliseCountsParams } from '@/lib/counts' +import { readCountState, setCountState, encodeCountState } from '@/lib/queryState' +import { postDefinition } from '@/test/fixtures/counts' +import { renderWithProviders } from '@/test/render' + +vi.mock ('@/lib/api', async original => ({ + ...await original (), apiGet: vi.fn ().mockResolvedValue ([]) })) + +const appliedParams = (extra: Record = {}) => normaliseCountsParams ('posts', + new URLSearchParams ({ mode: 'details', timezone: 'UTC', ...extra }), postDefinition) + +const renderControls = (params: URLSearchParams, labelsParams = params.toString ()) => { + const execute = vi.fn () + const result = renderWithProviders ( + + + , { route: '/ja/counts/posts' }) + return { ...result, execute } +} +const submit = () => fireEvent.submit ( + screen.getByRole ('button', { name: '集計実行' }).closest ('form')!) + + +describe ('Counts filter draft', () => { + it ('uses labels only for the executed condition set', () => { + const params = appliedParams () + setCountState (params, 'conditions', encodeCountState ('conditions', [ + { field: 'tags', operator: 'axis_eq', value: 'internal:7' }])) + const { execute } = renderControls (params) + expect (screen.getByText ('Resolved tag')).toBeInTheDocument () + fireEvent.change (screen.getByRole ('textbox', { name: 'Id.' }), { target: { value: '9' } }) + expect (screen.queryByText ('Resolved tag')).not.toBeInTheDocument () + expect (screen.getByText ('…')).toBeInTheDocument () + expect (screen.queryByText ('internal:7')).not.toBeInTheDocument () + submit () + expect (readCountState (execute.mock.lastCall![0], 'conditions', [])) + .toEqual ([{ field: 'tags', operator: 'axis_eq', value: 'internal:7' }]) + }) + + it ('removes a visible drill condition from the submitted URL state', () => { + const params = appliedParams () + setCountState (params, 'conditions', encodeCountState ('conditions', [ + { field: 'created_at', grain: 'month', operator: 'axis_eq', value: '2026-10' }])) + const { execute } = renderControls (params) + expect (screen.getByText (/令和8年10月/)).toBeInTheDocument () + fireEvent.click (screen.getByRole ('button', { name: '削除' })) + submit () + expect (readCountState (execute.mock.lastCall![0], 'conditions', [])).toEqual ([]) + }) + + it ('opens the existing add dialogue without inserting a condition immediately', () => { + const { execute } = renderControls (appliedParams ()) + fireEvent.click (screen.getByRole ('button', { name: '追加' })) + expect (screen.getByRole ('dialog')).toBeInTheDocument () + expect (execute).not.toHaveBeenCalled () + expect (document.querySelectorAll ('[role="dialog"] select.h-11').length).toBeGreaterThan (0) + expect (screen.queryByText ('pattern.input')).not.toBeInTheDocument () + }) + + it ('submits only the edited tag names', () => { + const { execute } = renderControls (appliedParams ({ tags: 'alpha beta' })) + const tags = screen.getByDisplayValue ('alpha beta') + fireEvent.change (tags, { target: { value: 'not:beta gamma alpha' } }) + submit () + expect (execute.mock.lastCall![0].get ('tags')).toBe ('not:beta gamma alpha') + fireEvent.change (tags, { target: { value: 'gamma' } }) + submit () + expect (execute.mock.lastCall![0].get ('tags')).toBe ('gamma') + }) + +}) + + +const FieldTree = () => { + const [params, setParams] = useState (() => appliedParams ({ columns: 'tags', + children: 'tags.name_ja' })) + return + setParams (previous => { + const next = new URLSearchParams (previous) + next.set (key, value) + return next + })}/> + {params.toString ()} + +} + +it ('derives mixed parent selection and permits selecting all children again after clearing', () => { + renderWithProviders (, { route: '/ja/counts/posts' }) + const parent = screen.getByRole ('checkbox', { name: 'タグ' }) + expect (parent).toHaveAttribute ('aria-checked', 'mixed') + expect (parent).toBeEnabled () + fireEvent.click (parent) + expect (parent).toBeChecked () + fireEvent.click (parent) + expect (parent).not.toBeChecked () + expect (parent).toBeEnabled () + fireEvent.click (parent) + expect (parent).toBeChecked () + const params = new URLSearchParams (screen.getByTestId ('fields').textContent ?? '') + expect (params.get ('children')?.split (' ')).toEqual ([ + 'tags.name_ja', 'tags.name_ain', 'tags.aliases_ja', 'tags.aliases_ain']) + expect (params.get ('columns')).toBe ('tags') +}) + + +it ('offers row numbers only in details and keeps limit in display settings', () => { + const view = renderControls (appliedParams ({ mode: 'summary', row_numbers: '1' })) + expect (screen.queryByRole ('checkbox', { name: '行番号' })).not.toBeInTheDocument () + const limit = screen.getByRole ('combobox', { name: '表示件数' }) + expect (Array.from (limit.querySelectorAll ('option')).map (option => option.value)) + .toEqual (['20', '50', '100']) + expect (limit).toHaveValue ('100') + expect (limit.querySelector ('span')).toBeNull () + view.unmount () + renderControls (appliedParams ({ row_numbers: '1' })) + expect (screen.getByRole ('checkbox', { name: '行番号' })).toBeChecked () +}) diff --git a/frontend/src/components/counts/CountsControls.tsx b/frontend/src/components/counts/CountsControls.tsx new file mode 100644 index 0000000..3ca3fe9 --- /dev/null +++ b/frontend/src/components/counts/CountsControls.tsx @@ -0,0 +1,281 @@ +import { useEffect, useId, useRef, useState } from 'react' + +import CountsAdditionalConditions from '@/components/counts/CountsAdditionalConditions' +import CountsAxes from '@/components/counts/CountsAxes' +import CountsFieldTree from '@/components/counts/CountsFieldTree' +import AnimatedCollapse from '@/components/common/AnimatedCollapse' +import DateTimeField from '@/components/common/DateTimeField' +import FormField from '@/components/common/FormField' +import HelpPopover from '@/components/common/HelpPopover' +import TagInput from '@/components/common/TagInput' +import { Button } from '@/components/ui/button' +import { countsRequestParams, countsUseLatest, fieldsForCounts, normaliseCountsParams, readCountSort, + selectedCountChildren, selectedCountColumns } from '@/lib/counts' +import { countsNestedIsScalar } from '@/lib/countsDisplay' +import { useCountsDefinition } from '@/lib/countsDefinitions' +import { useClientMotion } from '@/lib/useClientMotion' +import { inputClass } from '@/lib/utils' +import { useUITranslation } from '@/lib/useUITranslation' + +import { readCountState, setCountState } from '@/lib/queryState' + +import type { ReactNode } from 'react' + +import type { CountsResource, CountsResult } from '@/lib/counts' + + +const CountsControls = ( + { resource, applied, open, setOpen, execute, displayJSON, displayRowNumbers, + displayThumbnail, reorder, actions, conditionLabels, conditionLabelsParams, + pending = false }: { + resource: CountsResource; applied: URLSearchParams; open: boolean + conditionLabels?: CountsResult['conditionLabels'] + conditionLabelsParams?: string + pending?: boolean + setOpen: (open: boolean) => void + execute: (params: URLSearchParams) => void + displayJSON: (enabled: boolean) => void + displayRowNumbers: (enabled: boolean) => void + displayThumbnail: (enabled: boolean) => void + reorder: (value: string) => void; actions?: ReactNode }, +) => { + const { label, message } = useUITranslation ('counts') + const definition = useCountsDefinition () + const id = useId () + const motion = useClientMotion () + const [draft, setDraft] = useState (() => new URLSearchParams (applied)) + const labelParams = conditionLabelsParams == null + ? null : new URLSearchParams (conditionLabelsParams) + const thumbnail = selectedCountColumns (resource, applied, definition).includes ('thumbnail') + const previousApplied = useRef ({ resource, params: new URLSearchParams (applied) }) + const appliedState = applied.toString () + const mode = draft.get ('mode') ?? 'summary' + const columns = selectedCountColumns (resource, draft, definition) + const details = mode === 'details' + const nestedFlg = details && fieldsForCounts (resource, definition).some (field => + columns.includes (field.key) && field.children != null + && !countsNestedIsScalar (field.children, field.key, draft) + && field.children.some (child => selectedCountChildren (draft) + .has (`${field.key}.${child.key}`))) + useEffect (() => { + const nextApplied = new URLSearchParams (appliedState) + const previous = previousApplied.current + previousApplied.current = { resource, params: nextApplied } + setDraft (draft => { + if (previous.resource !== resource + || previous.params.get ('definition') !== nextApplied.get ('definition')) + return nextApplied + const keys = new Set ([...previous.params.keys (), ...nextApplied.keys ()]) + const changed = [...keys].filter (key => + previous.params.get (key) !== nextApplied.get (key)) + const before = selectedCountColumns (resource, previous.params, definition) + const after = selectedCountColumns (resource, nextApplied, definition) + const selectionChanged = before.filter (column => column !== 'thumbnail').join (' ') + !== after.filter (column => column !== 'thumbnail').join (' ') + const internal = ['execution_at', 'timezone', 'order', 'nested_json', 'row_numbers'] + if (changed.some (key => key === 'columns' ? selectionChanged : !internal.includes (key))) + return nextApplied + // Internal URL enrichment and immediate display changes preserve pending edits. + const next = new URLSearchParams (draft) + for (const key of changed) + { + if (key === 'columns') + { + const columns = selectedCountColumns (resource, draft, definition) + .filter (column => column !== 'thumbnail') + next.set ('columns', (after.includes ('thumbnail') + ? ['thumbnail', ...columns] : columns).join (' ')) + continue + } + next.delete (key) + for (const value of nextApplied.getAll (key)) next.append (key, value) + } + return next.toString () === draft.toString () ? draft : next + }) + }, [appliedState, resource, definition]) + const history = draft.get ('definition') === 'history' + const update = (key: string, value: string) => setDraft (previous => { + const next = new URLSearchParams (previous) + setCountState (next, key, value) + if (key === 'limit') next.delete ('page') + if (next.toString () === previous.toString ()) return previous + if (['rows', 'cols'].includes (key) && next.get ('sort') !== '') next.delete ('sort') + if (key !== 'mode') return next + const resolved = normaliseCountsParams (resource, next, definition) + const axes = [...readCountState (resolved, 'rows', []), + ...readCountState (resolved, 'cols', [])] + const allowed = value === 'summary' + ? ['count', ...axes.map ((_, index) => `a${index}`)] + : fieldsForCounts (resource, definition).map (field => field.key) + if (readCountSort (resolved.get ('sort')).some (sort => !allowed.includes (sort.key) + && !(value === 'summary' && /^m[0-9]{1,2}$/.test (sort.key)))) + next.delete ('sort') + return normaliseCountsParams (resource, next, definition) + }) + return
{ + event.preventDefault () + if (pending) return + const next = new URLSearchParams (draft) + next.delete ('page') + execute (normaliseCountsParams (resource, next, definition)) + }}> + {!history &&
+
+ {label ('referenceAt')} + {label ('referenceHelp')} +
+ update ('at', value ?? '')}/> + +
+
+
} +
+ + + + {actions} +
+
+ {label ('displaySettings')} + + {details && } + {details && resource === 'posts' && } + {nestedFlg && } +
+ +
+
+
+ {mode === 'summary' ? + : <> + {label ('output')} + { + update (key, value) + if (key === 'order') reorder (value) + }}/> + } +
+
+
+
+ {label ('filters')} + {history &&
+ update ('from', value ?? '')}/> + {label ('periodSeparator')} + update ('through', value ?? '')}/> +
} +
+ {['resource_id', 'user_id'].map (key => + {() => update (key, event.target.value)}/>} + )} +
+ {resource === 'posts' && + {() =>
+ update ('tags', value)}/> + + {draft.get ('tags') && <> + + } +
} + } + +
+
+
+ +
+} + + +export default CountsControls diff --git a/frontend/src/components/counts/CountsExportProgress.test.tsx b/frontend/src/components/counts/CountsExportProgress.test.tsx new file mode 100644 index 0000000..923fdf2 --- /dev/null +++ b/frontend/src/components/counts/CountsExportProgress.test.tsx @@ -0,0 +1,198 @@ +import { act, cleanup, screen } from '@testing-library/react' +import { StrictMode } from 'react' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +import CountsExportProgress from '@/components/counts/CountsExportProgress' +import { renderWithProviders } from '@/test/render' + +const api = vi.hoisted (() => ({ + apiPost: vi.fn (), apiGet: vi.fn (), apiDelete: vi.fn (), startAPIDownload: vi.fn () })) +vi.mock ('@/lib/api', async original => ({ + ...await original (), ...api })) + +const ready = { status: 'ready', processed: 5, total: 5, token: 'file' } +const receipt = { downloadAttempt: 'attempt', downloadStatus: 'pending', token: 'file' } +const controls = () => ({ close: vi.fn (), setActions: vi.fn (), confirm: vi.fn () }) +const flush = async (milliseconds = 0) => act (async () => { + await vi.advanceTimersByTimeAsync (milliseconds) +}) + +beforeEach (() => { + vi.useFakeTimers () + Object.values (api).forEach (mock => mock.mockReset ()) + api.apiPost.mockImplementation (async (path: string) => + path.endsWith ('/download') ? receipt : { job: 'job' }) + api.apiGet.mockResolvedValue (ready) + api.apiDelete.mockResolvedValue ({ status: 'cancelled', processed: 5, total: 5 }) + api.startAPIDownload.mockReturnValue (vi.fn ()) +}) +afterEach (() => { cleanup (); vi.useRealTimers () }) + +const renderProgress = (format = 'csv') => { + const actions = controls () + const handle = { cancel: async () => false } + const element = (next = actions) => + + + return { ...renderWithProviders (element ()), actions, handle, element } +} + +describe ('Counts native download acknowledgement', () => { + it.each (['csv', 'json']) ('closes %s after the matching GET acknowledgement', async format => { + api.apiGet.mockResolvedValueOnce (ready).mockResolvedValue ({ ...ready, + downloadAttempt: 'attempt', downloadStatus: 'started' }) + const view = renderProgress (format) + await flush () + expect (api.apiPost).toHaveBeenCalledTimes (2) + expect (api.startAPIDownload).toHaveBeenCalledTimes (1) + expect (view.actions.close).not.toHaveBeenCalled () + const query = api.startAPIDownload.mock.calls[0][3] as URLSearchParams + expect (Object.fromEntries (query)).toEqual ({ job: 'job', attempt: 'attempt' }) + const next = controls () + view.rerender (view.element (next)) + await flush (800) + expect (next.close).toHaveBeenCalledTimes (1) + expect (api.startAPIDownload).toHaveBeenCalledTimes (1) + }) + + it ('times out independently of pending status requests', async () => { + api.apiGet.mockResolvedValueOnce (ready).mockImplementation (() => new Promise (() => {})) + const view = renderProgress () + await flush () + await flush (30_000) + expect (screen.getByRole ('alert')).toHaveTextContent ('タイムアウト') + expect (view.actions.close).not.toHaveBeenCalled () + expect (api.startAPIDownload.mock.results[0].value).toHaveBeenCalled () + }) + + it ('retains the modal after acceptance failure without repeated POSTs', async () => { + api.apiPost.mockResolvedValueOnce ({ job: 'job' }) + .mockRejectedValueOnce (new Error ('offline')) + const view = renderProgress () + await flush () + expect (screen.getByRole ('alert')).toHaveTextContent ('開始できません') + await flush (40_000) + expect (api.apiPost).toHaveBeenCalledTimes (2) + expect (api.startAPIDownload).not.toHaveBeenCalled () + expect (view.actions.close).not.toHaveBeenCalled () + }) + + it ('retries a temporary status transport failure without a second native GET', async () => { + api.apiGet.mockResolvedValueOnce (ready).mockRejectedValueOnce (new Error ('offline')) + .mockResolvedValue ({ ...ready, downloadAttempt: 'attempt', downloadStatus: 'started' }) + const view = renderProgress () + await flush () + await flush (2800) + expect (view.actions.close).toHaveBeenCalledTimes (1) + expect (api.startAPIDownload).toHaveBeenCalledTimes (1) + }) + + it ('cancels an acceptance in flight and never starts the resulting receipt', async () => { + let resolve!: (value: typeof receipt) => void + api.apiPost.mockResolvedValueOnce ({ job: 'job' }).mockImplementationOnce ( + () => new Promise (complete => { resolve = complete })) + const view = renderProgress () + await flush () + await act (async () => { expect (await view.handle.cancel ()).toBe (true) }) + await act (async () => { resolve (receipt) }) + await flush (1000) + expect (api.startAPIDownload).not.toHaveBeenCalled () + expect (view.actions.close).not.toHaveBeenCalled () + }) + + it ('releases an unmounted iframe and ignores late acknowledgements', async () => { + const view = renderProgress () + await flush () + const release = api.startAPIDownload.mock.results[0].value + view.unmount () + await flush () + expect (release).toHaveBeenCalledTimes (1) + await flush (40_000) + expect (view.actions.close).not.toHaveBeenCalled () + expect (api.startAPIDownload).toHaveBeenCalledTimes (1) + }) + + it ('keeps authentication expiry and iframe errors visible', async () => { + const view = renderProgress () + await flush () + act (() => { api.startAPIDownload.mock.calls[0][2] () }) + expect (screen.getByRole ('alert')).toHaveTextContent ('開始できません') + await flush (40_000) + expect (view.actions.close).not.toHaveBeenCalled () + }) + it ('shows the worker-specific reason instead of a download-success message', async () => { + api.apiPost.mockRejectedValueOnce ({ isAxiosError: true, + response: { status: 503, data: { error: 'export_worker_unavailable' } } }) + const view = renderProgress () + await flush () + expect (screen.getByRole ('alert')).toHaveTextContent ('ワーカーが稼働していません') + expect (view.actions.close).not.toHaveBeenCalled () + expect (await view.handle.cancel ()).toBe (true) + }) + + it ('shows revoked authority and does not close after a successful acceptance', async () => { + api.apiGet.mockResolvedValueOnce (ready).mockRejectedValue ({ isAxiosError: true, + response: { status: 404 } }) + const view = renderProgress () + await flush () + await flush (800) + expect (screen.getByRole ('alert')).toHaveTextContent ('認可が失われたか') + expect (view.actions.close).not.toHaveBeenCalled () + }) + + it ('stops after cancellation failure and permits an explicit cancellation retry', async () => { + const view = renderProgress () + await flush () + api.apiDelete.mockRejectedValueOnce (new Error ('offline')) + await act (async () => { expect (await view.handle.cancel ()).toBe (false) }) + const requests = api.apiGet.mock.calls.length + await flush (40_000) + expect (api.apiGet).toHaveBeenCalledTimes (requests) + expect (screen.getByRole ('alert')).toBeInTheDocument () + await act (async () => { expect (await view.handle.cancel ()).toBe (true) }) + expect (api.apiDelete).toHaveBeenCalledTimes (2) + expect (api.startAPIDownload).toHaveBeenCalledTimes (1) + }) + + it ('ignores a different attempt even when it reports started', async () => { + api.apiGet.mockResolvedValueOnce (ready).mockResolvedValueOnce ({ ...ready, + downloadAttempt: 'another', downloadStatus: 'started' }).mockResolvedValue ({ ...ready, + downloadAttempt: 'attempt', downloadStatus: 'started' }) + const view = renderProgress () + await flush () + await flush (800) + expect (view.actions.close).not.toHaveBeenCalled () + await flush (800) + expect (view.actions.close).toHaveBeenCalledTimes (1) + }) + + it ('keeps the created job available for cancellation retry during initial registration', async () => { + let resolve!: (value: { job: string }) => void + api.apiPost.mockImplementationOnce (() => new Promise (complete => { resolve = complete })) + api.apiDelete.mockRejectedValueOnce (new Error ('offline')) + const view = renderProgress () + await flush () + let pending!: Promise + act (() => { pending = view.handle.cancel () }) + await act (async () => { + resolve ({ job: 'created-during-cancel' }) + expect (await pending).toBe (false) + }) + await act (async () => { expect (await view.handle.cancel ()).toBe (true) }) + expect (api.apiDelete.mock.calls.map (call => call[0])).toEqual ([ + '/counts/export/jobs/created-during-cancel', '/counts/export/jobs/created-during-cancel']) + expect (api.startAPIDownload).not.toHaveBeenCalled () + }) + + it ('stops immediately if native download navigation cannot be created', async () => { + api.startAPIDownload.mockImplementationOnce (() => { throw new Error ('navigation blocked') }) + const view = renderProgress () + await flush () + expect (screen.getByRole ('alert')).toHaveTextContent ('開始できません') + await flush (40_000) + expect (api.apiGet).toHaveBeenCalledTimes (1) + expect (view.actions.close).not.toHaveBeenCalled () + }) + +}) diff --git a/frontend/src/components/counts/CountsExportProgress.tsx b/frontend/src/components/counts/CountsExportProgress.tsx new file mode 100644 index 0000000..240b26b --- /dev/null +++ b/frontend/src/components/counts/CountsExportProgress.tsx @@ -0,0 +1,279 @@ +import { useEffect, useRef, useState } from 'react' + +import LocalisedText from '@/components/LocalisedText' +import { apiDelete, apiGet, apiPost, isApiError, startAPIDownload } from '@/lib/api' +import { countsErrorMessage } from '@/lib/countsDisplay' +import { useClientMotion } from '@/lib/useClientMotion' +import { useUITranslation } from '@/lib/useUITranslation' +import { cn } from '@/lib/utils' + +import type { DialogueFormControls } from '@/lib/dialogues/useDialogue' +import type { LocaleCode } from '@/lib/locales' + +export type CountsExportHandle = { cancel: () => Promise } +type ExportStatus = { + status: 'starting' | 'queued' | 'running' | 'cancelling' | 'cancelled' | 'ready' | 'failed' + processed: number; total: number | null; token?: string + downloadAttempt?: string; downloadStatus?: string; error?: string + phase?: 'queued' | 'aggregating' | 'writing'; heartbeatAt?: number; updatedAt?: number } + + +const CountsExportProgress = ( + { payload, locale, handle, controls }: { + payload: Record; locale: LocaleCode + handle: CountsExportHandle; controls: DialogueFormControls }, +) => { + const { label, message } = useUITranslation ('counts') + const motion = useClientMotion () + const [state, setState] = useState ({ + status: 'starting', processed: 0, total: null }) + const [cancelError, setCancelError] = useState (null) + const [failure, setFailure] = useState (null) + const [downloadError, setDownloadError] = useState (null) + // One mounted dialogue owns one immutable job, even if its parent rerenders. + const initial = useRef ({ payload, locale }) + const actions = useRef (controls) + actions.current = controls + const terminal = useRef (false) + const finished = useRef (false) + const cancelAction = useRef<() => Promise> (async () => false) + const cancelled = useRef (false) + const generation = useRef (0) + const job = useRef (null) + const creation = useRef | null> (null) + const registration = useRef | null> (null) + const attempt = useRef (null) + const downloadDeadline = useRef (0) + const releaseDownload = useRef<(() => void) | null> (null) + const cancellation = useRef | null> (null) + useEffect (() => { handle.cancel = () => cancelAction.current () }, [handle]) + useEffect (() => { + const epoch = ++generation.current + let active = true + let lastResponse = Date.now () + let timer: ReturnType | null = null + let acknowledgementTimer: ReturnType | null = null + let request: AbortController | null = null + const current = () => active && generation.current === epoch + const stop = (reason: string) => { + if (!current () || terminal.current) return + terminal.current = true + if (timer != null) clearTimeout (timer) + if (acknowledgementTimer != null) clearTimeout (acknowledgementTimer) + request?.abort () + releaseDownload.current?.() + releaseDownload.current = null + setDownloadError (reason) + } + const cancel = (): Promise => { + if (finished.current || (terminal.current && job.current == null)) + return Promise.resolve (true) + cancelled.current = true + terminal.current = true + if (timer != null) clearTimeout (timer) + if (acknowledgementTimer != null) clearTimeout (acknowledgementTimer) + request?.abort () + releaseDownload.current?.() + releaseDownload.current = null + cancellation.current ??= (async () => { + const controller = new AbortController () + const deadline = setTimeout (() => controller.abort (), 15_000) + try + { + const created = await creation.current + if (created == null) return false + job.current = created.job + const result = await apiDelete ( + `/counts/export/jobs/${encodeURIComponent (created.job)}`, { + locale: initial.current.locale, signal: controller.signal }) + if (current ()) + { + setState (result) + setCancelError (null) + finished.current = ['cancelled', 'failed'].includes (result.status) + terminal.current = finished.current + if (!terminal.current) + { + if (timer != null) clearTimeout (timer) + timer = setTimeout (() => poll (created.job), 800) + } + } + return ['cancelled', 'failed'].includes (result.status) + } + catch (error) + { + if (isApiError (error) && error.response?.status === 404) return true + if (current ()) setCancelError (error) + return false + } + finally + { + clearTimeout (deadline) + cancellation.current = null + } + }) () + return cancellation.current + } + cancelAction.current = cancel + const armDeadline = () => { + if (downloadDeadline.current === 0) return + acknowledgementTimer = setTimeout (() => stop ('timeout'), + Math.max (0, downloadDeadline.current - Date.now ())) + } + const poll = async (token: string) => { + if (!current () || terminal.current) return + const controller = new AbortController () + request = controller + const deadline = setTimeout (() => controller.abort (), 15_000) + try + { + const result = await apiGet ( + `/counts/export/jobs/${encodeURIComponent (token)}`, { + locale: initial.current.locale, signal: controller.signal }) + if (!current () || terminal.current) return + lastResponse = Date.now () + setState (result) + if (cancelled.current) + { + if (await cancel ()) return + } + else if (result.status === 'ready') + { + if (result.token == null) { stop ('failed'); return } + if (attempt.current == null) + { + registration.current ??= apiPost ( + `/counts/export/jobs/${encodeURIComponent (token)}/download`, {}, { + locale: initial.current.locale, signal: controller.signal }) + const receipt = await registration.current + if (!current () || terminal.current || cancelled.current) return + if (receipt.downloadAttempt == null) { stop ('failed'); return } + attempt.current = receipt.downloadAttempt + downloadDeadline.current = Date.now () + 30_000 + armDeadline () + try + { + releaseDownload.current = startAPIDownload ('/counts/export', result.token, + () => stop ('failed'), new URLSearchParams ({ + job: token, attempt: receipt.downloadAttempt })) + } + catch + { + stop ('failed') + return + } + } + if (result.downloadAttempt === attempt.current) + { + if (result.downloadStatus === 'started') + { + terminal.current = true + finished.current = true + if (acknowledgementTimer != null) clearTimeout (acknowledgementTimer) + // Rack yielded a chunk; browser receipt and saving are not confirmed. + releaseDownload.current = null + actions.current.close () + return + } + if (['failed', 'unavailable', 'timeout'].includes (result.downloadStatus ?? '')) + { stop (result.downloadStatus!); return } + } + } + else if (['failed', 'cancelled'].includes (result.status)) + { + finished.current = true + terminal.current = true + return + } + if (timer != null) clearTimeout (timer) + timer = setTimeout (() => poll (token), 800) + } + catch (error) + { + if (!current () || terminal.current) return + const status = isApiError (error) ? error.response?.status : null + if ([400, 401, 403, 422].includes (status ?? 0)) setFailure (error) + if (registration.current != null && attempt.current == null) + { stop ('failed'); return } + if (status != null && status >= 400 && status < 500) + { stop ('unavailable'); return } + if (Date.now () - lastResponse >= 30_000) + { stop ('timeout'); return } + timer = setTimeout (() => poll (token), 2000) + } + finally + { + clearTimeout (deadline) + if (request === controller) request = null + } + } + if (creation.current == null) + { + const controller = new AbortController () + const deadline = setTimeout (() => controller.abort (), 15_000) + creation.current = apiPost<{ job: string }> ( + '/counts/export/jobs', initial.current.payload, { + locale: initial.current.locale, signal: controller.signal }) + .finally (() => clearTimeout (deadline)) + } + armDeadline () + void creation.current.then (result => { + if (!current () || terminal.current) return + job.current = result.job + void poll (result.job) + }).catch (error => { + if (current ()) setFailure (error) + stop ('failed') + }) + return () => { + active = false + if (timer != null) clearTimeout (timer) + if (acknowledgementTimer != null) clearTimeout (acknowledgementTimer) + request?.abort () + // StrictMode immediately installs another effect; real unmount releases the frame. + queueMicrotask (() => { + if (generation.current !== epoch) return + releaseDownload.current?.() + releaseDownload.current = null + }) + } + }, []) + useEffect (() => { controls.setActions ([]) }, [controls]) + const percent = state.total == null ? null : (state.total === 0 ? 100 + : Math.min (100, Math.floor (state.processed / state.total * 100))) + const busy = downloadError == null && cancelError == null + && !['failed', 'cancelled'].includes (state.status) + const phaseKey = (downloadError != null || cancelError != null) ? 'failed' + : (state.status === 'running' ? state.phase ?? 'running' + : (state.status === 'ready' ? 'writing' : state.status)) + const phase = message (`exportProgress.${phaseKey}`) + const failureMessage = countsErrorMessage (failure ?? state, locale) + const cancellationMessage = countsErrorMessage (cancelError, locale) + return
+

{label ('exportProgress.format', { + format: message (`exportFormats.${payload.format}`).text })}

+

+ {busy &&

+ +

{percent == null + ? label ('exportProgress.processed', { count: state.processed }) + : label ('exportProgress.measured', { + processed: state.processed, total: state.total ?? 0, percent })}

+ {(downloadError != null || state.status === 'failed') &&

+ {failure != null || state.status === 'failed' + ? failureMessage.text + : label (`downloadErrors.${downloadError}`)} + {downloadError != null && <>
{label ('errors.stopped')}}

} + {cancelError != null &&

+ {cancellationMessage.text}

} +
+} + +export default CountsExportProgress diff --git a/frontend/src/components/counts/CountsFieldTree.tsx b/frontend/src/components/counts/CountsFieldTree.tsx new file mode 100644 index 0000000..22e9c3e --- /dev/null +++ b/frontend/src/components/counts/CountsFieldTree.tsx @@ -0,0 +1,137 @@ +import { useId, useLayoutEffect, useRef, useState } from 'react' + +import AnimatedCollapse from '@/components/common/AnimatedCollapse' +import SortableList from '@/components/common/SortableList' +import { countChildPaths, fieldsForCounts, normaliseCountsParams, orderedCountFields, + selectedCountChildren, selectedCountColumns, selectedCountFields } from '@/lib/counts' +import { useCountsDefinition } from '@/lib/countsDefinitions' +import { useClientMotion } from '@/lib/useClientMotion' +import { useUITranslation } from '@/lib/useUITranslation' + +import type { CountField, CountsResource } from '@/lib/counts' + +type BranchProps = { + field: CountField; path: string; enabled: boolean + selected: Set; params: URLSearchParams + select: (path: string, checked: boolean) => void + update: (key: string, value: string) => void } + + +const FieldBranch = ({ field, path, enabled, selected, params, select, update }: BranchProps) => { + const { label, message } = useUITranslation ('counts') + const [open, setOpen] = useState (false) + const id = useId () + const motion = useClientMotion () + const input = useRef (null) + const leaves = countChildPaths (field.children ?? [], path, true) + const count = leaves.filter (key => selected.has (key)).length + const checked = field.children == null ? selected.has (path) + : count > 0 && count === leaves.length + const partial = count > 0 && count < leaves.length + useLayoutEffect (() => { + if (input.current != null) input.current.indeterminate = partial + }, [partial]) + return <> +
+ + {field.children != null && } +
+ {field.children != null && +
+ +
+ } + +} + + +const FieldGroup = ({ fields, path, params, enabled, selected, select, update }: { + fields: CountField[]; path: string; params: URLSearchParams + enabled: boolean; selected: Set; select: BranchProps['select'] + update: BranchProps['update'] }, +) => { + const { message } = useUITranslation ('counts') + const ordered = orderedCountFields (fields, params, path) + const prefix = path ? `${path}.` : '' + return field.key) }} + fixed={path === '' ? ['thumbnail'] : []} + name={key => message (fields.find (field => field.key === key)!.label, + fields.find (field => field.key === key)!.labelValues)} + handleName={key => message ('reorder', { + column: message (fields.find (field => field.key === key)!.label, + fields.find (field => field.key === key)!.labelValues).text })} + renderGroup={(_, content) => content} + onChange={groups => { + const current = fields.map (field => `${prefix}${field.key}`) + const other = (params.get ('order') ?? '').split (' ') + .filter (key => key && !current.includes (key)) + update ('order', [...other, ...groups.items.map (key => `${prefix}${key}`)].join (' ')) + }} renderItem={key => { + const field = fields.find (field => field.key === key)! + return + }}/> +} + + +const CountsFieldTree = ( + { resource, params, update, fields: suppliedFields, + selectionKey = 'columns', parent = '' }: { + resource: CountsResource + fields?: CountField[] + selectionKey?: string + parent?: string + params: URLSearchParams + update: (key: string, value: string) => void }, +) => { + const definition = useCountsDefinition () + const resolved = normaliseCountsParams (resource, params, definition) + const fields = suppliedFields ?? fieldsForCounts (resource, definition) + const roots = suppliedFields == null ? selectedCountColumns (resource, resolved, definition) + : selectedCountFields (fields, resolved, selectionKey, parent).map (field => field.key) + const selected = selectedCountChildren (resolved, fieldsForCounts (resource, definition)) + const rootPath = (key: string) => parent ? `${parent}.${key}` : key + for (const root of roots) selected.add (rootPath (root)) + const select = (path: string, checked: boolean) => { + const next = new Set (selected) + const paths = countChildPaths (fields, parent) + .filter (key => key === path || key.startsWith (`${path}.`)) + for (const key of paths) + { + if (checked) next.add (key) + else next.delete (key) + } + const nextParams = new URLSearchParams (resolved) + nextParams.set ('children', [...next].filter (key => key.includes ('.')).join (' ')) + const children = selectedCountChildren (nextParams, fieldsForCounts (resource, definition)) + update ('children', [...children].join (' ')) + update (selectionKey, fields.filter (field => field.children == null + ? next.has (rootPath (field.key)) : field.children.some (child => + children.has (`${rootPath (field.key)}.${child.key}`))) + .map (field => field.key).join (' ')) + } + const visibleFields = parent === '' + ? fields.filter (field => field.key !== 'thumbnail') : fields + return ( + <> + + ) +} + + +export default CountsFieldTree diff --git a/frontend/src/components/counts/CountsNestedTable.tsx b/frontend/src/components/counts/CountsNestedTable.tsx new file mode 100644 index 0000000..fe858a7 --- /dev/null +++ b/frontend/src/components/counts/CountsNestedTable.tsx @@ -0,0 +1,126 @@ +import LocalisedText from '@/components/LocalisedText' +import PrefetchLink from '@/components/PrefetchLink' +import CountsSortHeader from '@/components/counts/CountsSortHeader' +import CountsSelectableTable from '@/components/counts/CountsSelectableTable' +import { changeCountSort, readCountSort, selectedCountChildren, + writeCountSort, orderedCountFields } from '@/lib/counts' +import { countsFieldValue, countsLogicalName, countsNestedClipboardTable, countsNestedExport, + countsNestedOrder, countsNestedText, countsNestedIsScalar, countsNestedScalarText } + from '@/lib/countsDisplay' +import { useLocale } from '@/lib/useLocale' +import { useUITranslation } from '@/lib/useUITranslation' + +import type { CountField, CountRecord, CountsResource } from '@/lib/counts' + + +const CountsNestedTable = ( + { rows, fields, path, resource, title, params, update }: { + rows: CountRecord[] + fields: CountField[] + path: string + resource: CountsResource + title: { text: string; language: string } + params: URLSearchParams + update: (key: string, value: string) => void }, +) => { + const { message } = useUITranslation ('counts') + const locale = useLocale () + const selected = selectedCountChildren (params) + const columns = orderedCountFields (fields, params, path) + .filter (field => selected.has (`${path}.${field.key}`)) + const allSorts = readCountSort (params.get ('child_sort')) + const sorts = allSorts.filter (sort => sort.key.startsWith (`${path}/`)) + .map (sort => ({ ...sort, key: sort.key.slice (path.length + 1) })) + const ordered = countsNestedOrder (rows, path, params, locale) + const scalar = (row: CountRecord, field: string) => { + const value = countsFieldValue (row, field) + if (value == null) return null + if (field === 'tagId' || field === 'tagGroupId') + return + {String (value)} + if (field === 'id') + { + if (row.kind === 'external') return String (value) + const target = path === 'parents' ? resource : 'tags' + return + {String (value)} + } + const display = countsLogicalName (row, field, locale) + const metadata = fields.find (item => item.key === field) + const nameLocale = metadata?.nameLocale + if (nameLocale != null) + { + if (metadata?.multipleValues) + return {display.text} + return {display.text} + } + if (typeof value === 'boolean') + return {value ? '✓' : '✗'} + if (display.language != null) + return {display.text} + if (field === 'name' && typeof row.languageCode === 'string') + return {String (value)} + return display.text + } + if (countsNestedIsScalar (fields, path, params)) + return + {countsNestedScalarText (rows, fields, path, params, locale)} + return ( +
+ {params.get ('nested_json') === '1' ? ( + 
+	      {JSON.stringify (countsNestedExport (rows, fields, path, params, locale))}
+
) : (columns.length > 0 && rows.length > 0 && ( +
+ field.key)} + identity={`${path}:${params}`} name={title} + copyTable={() => countsNestedClipboardTable (rows, fields, path, params, locale)} + className="text-xs"> + + {columns.map (field => ( + { + const next = changeCountSort (sorts, key, additive) + .map (sort => ({ ...sort, key: `${path}/${sort.key}` })) + update ('child_sort', writeCountSort ([ + ...allSorts.filter (sort => !sort.key.startsWith (`${path}/`)), + ...next])) + }}/>))} + + {ordered.map ((row, index) => ( + + {columns.map (field => ( + + {field.children + ? + : scalar (row, field.key)} + ))} + ))} + +
))} +
) +} + + +export default CountsNestedTable diff --git a/frontend/src/components/counts/CountsResizeHandle.tsx b/frontend/src/components/counts/CountsResizeHandle.tsx new file mode 100644 index 0000000..62718f9 --- /dev/null +++ b/frontend/src/components/counts/CountsResizeHandle.tsx @@ -0,0 +1,93 @@ +import { useContext, useEffect, useRef, useState } from 'react' + +import { CountsTableScope, MAX_COUNT_WIDTH, MIN_COUNT_WIDTH, + useCountsTableLayout } from '@/lib/countsTableLayout' +import { cn } from '@/lib/utils' +import { useUITranslation } from '@/lib/useUITranslation' + + +const CountsResizeHandle = ({ field, label }: { + field: string; label: { text: string; language: string } }) => { + const family = useContext (CountsTableScope) + const layout = useCountsTableLayout () + const { message } = useUITranslation ('counts') + const [active, setActive] = useState (false) + const press = useRef<{ x: number; y: number; width: number; armed: boolean + timer?: ReturnType } | null> (null) + const saved = useRef<{ cursor: string; userSelect: string } | null> (null) + const restore = () => { + if (saved.current == null) return + document.body.style.cursor = saved.current.cursor + document.body.style.userSelect = saved.current.userSelect + saved.current = null + } + const finish = () => { + if (press.current?.timer != null) clearTimeout (press.current.timer) + press.current = null + setActive (false) + restore () + } + const arm = () => { + if (press.current == null) return + press.current.armed = true + saved.current = { cursor: document.body.style.cursor, + userSelect: document.body.style.userSelect } + document.body.style.cursor = 'col-resize' + document.body.style.userSelect = 'none' + setActive (true) + } + useEffect (() => () => { + if (press.current?.timer != null) clearTimeout (press.current.timer) + if (saved.current != null) + { + document.body.style.cursor = saved.current.cursor + document.body.style.userSelect = saved.current.userSelect + } + }, []) + const name = message ('resize.column', { column: label.text }) + return ( + { + event.stopPropagation () + if (!event.isPrimary || event.button !== 0) return + event.preventDefault () + finish () + event.currentTarget.setPointerCapture (event.pointerId) + press.current = { x: event.clientX, y: event.clientY, + width: layout.width (family, field), armed: false } + if (event.pointerType === 'mouse') arm () + else press.current.timer = setTimeout (arm, 400) + }} + onPointerMove={event => { + event.stopPropagation () + const state = press.current + if (state == null) return + if (state.armed) + layout.resize (family, field, state.width + event.clientX - state.x) + else if (Math.hypot (event.clientX - state.x, event.clientY - state.y) > 8) + finish () + }} + onPointerUp={finish} onPointerCancel={finish} onLostPointerCapture={finish} + onClick={event => { event.preventDefault (); event.stopPropagation () }} + onContextMenu={event => event.preventDefault ()} + onKeyDown={event => { + event.stopPropagation () + if (event.key !== 'ArrowLeft' && event.key !== 'ArrowRight') return + event.preventDefault () + layout.resize (family, field, layout.width (family, field) + + (event.key === 'ArrowLeft' ? -10 : 10)) + }}/>) +} + + +export default CountsResizeHandle diff --git a/frontend/src/components/counts/CountsResults.tsx b/frontend/src/components/counts/CountsResults.tsx new file mode 100644 index 0000000..793e3f0 --- /dev/null +++ b/frontend/src/components/counts/CountsResults.tsx @@ -0,0 +1,173 @@ +import SortableColumns from '@/components/common/SortableColumns' +import LocalisedText from '@/components/LocalisedText' +import PrefetchLink from '@/components/PrefetchLink' +import CountsRowNumber from '@/components/counts/CountsRowNumber' +import CountsNestedTable from '@/components/counts/CountsNestedTable' +import CountsSortHeader from '@/components/counts/CountsSortHeader' +import CountsResizeHandle from '@/components/counts/CountsResizeHandle' +import CountsSelectableTable from '@/components/counts/CountsSelectableTable' +import PostThumbnailPreview from '@/components/posts/PostThumbnailPreview' +import { changeCountSort, countColumns, countsRequestParams, fieldsForCounts, + readCountSort, selectedCountColumns, writeCountSort } from '@/lib/counts' +import { countsCellCopy, countsCellText, countsDetailClipboardTable, + countsImageURL, countsNestedRows } from '@/lib/countsDisplay' +import useDialogue from '@/lib/dialogues/useDialogue' +import { useLocale } from '@/lib/useLocale' +import { useCountsDefinition } from '@/lib/countsDefinitions' +import { useUITranslation } from '@/lib/useUITranslation' + +import type { CountColumn, CountsResource, + CountsResult, CountsVersion } from '@/lib/counts' + + +const CountsResults = ( + { resource, data, params, pending, update }: { + resource: CountsResource + data: CountsResult + params: URLSearchParams + pending: boolean + update: (key: string, value: string) => void }, +) => { + const definition = useCountsDefinition () + const { label, message } = useUITranslation ('counts') + const locale = useLocale () + const dialogue = useDialogue () + const columns = selectedCountColumns (resource, params, definition) + const rowNumbers = params.get ('row_numbers') === '1' + const fields = fieldsForCounts (resource, definition) + const sorts = readCountSort (params.get ('sort') ?? 'resource_id:asc version_no:asc') + const sort = (key: string, additive: boolean) => + update ('sort', writeCountSort (changeCountSort (sorts, key, additive))) + const cell = (row: CountsVersion, column: CountColumn) => { + const field = fields.find (field => field.key === column)! + const value = countsCellText (row, column, locale, params) + if (field.children != null) + { + return + } + if (field.nameLocale != null) + { + if (field.multipleValues) + return {value} + return {value} + } + switch (column) + { + case 'resource_id': + if (resource === 'nico_tags') return row.name ? ( + + {row.resourceId}) : row.resourceId + if (resource === 'tag_groups') return + {row.resourceId} + return + {row.resourceId} + case 'name': + return resource === 'nico_tags' && row.name ? ( + + {value}) : value + case 'thumbnail': + { + const url = countsImageURL (row.thumbnailBase) + return url == null ? null : ( + ) + } + case 'url': + case 'thumbnail_base': + return + {value} + case 'latest': + return + {value} + case 'user': + return row.createdByUser?.name ? value : ( + + {value}) + case 'event_type': + return + {value} + case 'category': + return + {value} + default: return value + } + } + return message (fields.find (field => field.key === key)!.label, + fields.find (field => field.key === key)!.labelValues)} + onChange={next => { + const all = fields.map (field => field.key) + const other = (params.get ('order') ?? '').split (' ').filter (key => !all.includes (key)) + update ('order', [...next, ...all.filter (key => !next.includes (key)), ...other].join (' ')) + }}>{columns => { + const tableColumns = rowNumbers ? ['row_number', ...columns] : columns + return ( +
+ {tableColumns.length === 0 ?

{label ('nested.noColumns')}

: ( + countsDetailClipboardTable (data.versions ?? [], + columns.map (column => fields.find (field => field.key === column)!), + params, locale, rowNumbers ? (data.page - 1) * data.limit + 1 : null)} + identity={`${resource}:${countsRequestParams (params)}:${tableColumns.join (' ')}:${ + (data.versions ?? []).map (row => row.id)}`} + className="text-left text-sm"> + + {rowNumbers && } + {columns.map (column => column === 'thumbnail' + ? + {label (countColumns[column])} + + + : field.key === column)!.label, + fields.find (field => field.key === column)!.labelValues)} + dragId={column} + sorts={sorts} change={sort} structured={fields.find ( + field => field.key === column)?.children != null}/>)} + + + {(data.versions ?? []).map ((row, index) => ( + + {rowNumbers && } + {columns.map (column => ( + field.key === column)!, params, locale)}> + {cell (row, column)}))} + ))} + + )} + {(data.versions?.length ?? 0) === 0 + &&

{label ('empty')}

} +
) + }} +} + + +export default CountsResults diff --git a/frontend/src/components/counts/CountsRowNumber.tsx b/frontend/src/components/counts/CountsRowNumber.tsx new file mode 100644 index 0000000..822eb1c --- /dev/null +++ b/frontend/src/components/counts/CountsRowNumber.tsx @@ -0,0 +1,17 @@ +import CountsResizeHandle from '@/components/counts/CountsResizeHandle' +import { useUITranslation } from '@/lib/useUITranslation' + + +const CountsRowNumber = ({ value, rowSpan }: { value?: number; rowSpan?: number }) => { + const { label, message } = useUITranslation ('counts') + if (value != null) return + {value} + return + {label ('rowNumberHeading')} + + +} + + +export default CountsRowNumber diff --git a/frontend/src/components/counts/CountsSelectableTable.tsx b/frontend/src/components/counts/CountsSelectableTable.tsx new file mode 100644 index 0000000..79663d9 --- /dev/null +++ b/frontend/src/components/counts/CountsSelectableTable.tsx @@ -0,0 +1,631 @@ +import { useCallback, useEffect, useId, useLayoutEffect, useMemo, useRef, useState } from 'react' +import { createPortal } from 'react-dom' + +import CountsCellMenu from '@/components/counts/CountsCellMenu' +import { countsClipboard, writeCountsClipboard } from '@/lib/countsClipboard' +import { CountsTableScope, useCountsTableLayout } from '@/lib/countsTableLayout' +import { cn } from '@/lib/utils' +import { useUITranslation } from '@/lib/useUITranslation' + +import type { ClipboardEvent, ReactNode } from 'react' + +import type { CountsClipboardTable } from '@/lib/countsClipboard' +import type { CountsMenuAction } from '@/components/counts/CountsCellMenu' + +type Point = { row: number; column: number } +type Selection = { identity: string; anchor: Point; end: Point } +type TextMode = { identity: string; cell: HTMLTableCellElement; nodes: Node[] } +type Caret = { cell: HTMLTableCellElement; left: number; top: number; height: number } +const interactive = 'a,button,input,select,textarea,summary,[contenteditable],[data-counts-resize]' + + +const cellGrid = (table: HTMLTableElement): HTMLTableCellElement[][] => { + const grid: HTMLTableCellElement[][] = [] + for (const [row, tr] of Array.from (table.tBodies[0]?.rows ?? []).entries ()) + { + grid[row] ??= [] + let column = 0 + for (const cell of Array.from (tr.cells)) + { + while (grid[row][column] != null) ++column + for (let y = 0; y < cell.rowSpan; ++y) + { + grid[row + y] ??= [] + for (let x = 0; x < cell.colSpan; ++x) grid[row + y][column + x] = cell + } + column += cell.colSpan + } + } + return grid +} + + +const cellAt = (table: HTMLTableElement, point: Point) => + cellGrid (table)[point.row]?.[point.column] + + +const scrollToCell = (table: HTMLTableElement, point: Point, previous: Point) => { + const cell = cellAt (table, point) + const container = table.parentElement?.closest ( + '[data-counts-scroll],[data-counts-results]') + if (cell == null || container == null) return + const viewport = container.getBoundingClientRect () + const top = viewport.top + container.clientTop + const left = viewport.left + container.clientLeft + const bottom = top + container.clientHeight + const right = left + container.clientWidth + const header = table.tHead?.getBoundingClientRect () + const visibleTop = header == null ? top : Math.max (top, Math.min (bottom, header.bottom)) + const rect = cell.getBoundingClientRect () + const lastRow = (table.tBodies[0]?.rows.length ?? 1) - 1 + const verticalEdge = (point.row === 0 || point.row < previous.row) ? 'start' + : ((point.row === lastRow || point.row > previous.row) ? 'end' + : (Math.abs (rect.top - visibleTop) <= Math.abs (rect.bottom - bottom) ? 'start' : 'end')) + const offset = ( + start: number, end: number, minimum: number, maximum: number, + edge?: 'start' | 'end', + ) => { + if (maximum <= minimum) return 0 + const oversized = end - start > maximum - minimum + // Vertical merged cells follow the destination and movement direction. + // Horizontal scrolling retains its existing nearest-edge behaviour. + if (oversized && edge != null) return edge === 'start' ? start - minimum : end - maximum + if (start < minimum && end > maximum) return 0 + if (start < minimum) return oversized ? end - maximum : start - minimum + if (end > maximum) return oversized ? start - minimum : end - maximum + return 0 + } + const vertical = offset (rect.top, rect.bottom, visibleTop, bottom, verticalEdge) + const horizontal = offset (rect.left, rect.right, left, right) + if (vertical !== 0) container.scrollTop += vertical + if (horizontal !== 0) container.scrollLeft += horizontal +} + + +const CountsSelectableTable = ( + { family, columns, identity, children, className, name, copyTable }: { + family: string + columns: string[] + identity: string + children: ReactNode + className?: string + name?: { text: string; language: string } + copyTable?: () => CountsClipboardTable }, +) => { + const id = useId () + const root = useRef (null) + const [textMode, setTextMode] = useState (null) + const textCell = textMode?.identity === identity ? textMode.cell : null + const [caret, setCaret] = useState (null) + const mousePress = useRef<{ x: number; y: number } | null> (null) + const dragging = useRef (false) + const touchArmed = useRef (false) + const touchPress = useRef<{ x: number; y: number + timer: ReturnType } | null> (null) + const [menu, setMenu] = useState<{ x: number; y: number; point: Point + identity: string } | null> (null) + const suppressClick = useRef (false) + const [stored, setStored] = useState (null) + const selection = stored?.identity === identity ? stored : null + const layout = useCountsTableLayout () + const { label, message } = useUITranslation ('counts') + const clearTextSelection = useCallback (() => { + setTextMode (null) + setCaret (null) + }, []) + useLayoutEffect (() => { + if (textMode == null) return + const { cell, nodes } = textMode + const table = root.current + if (table == null || textMode.identity !== identity || !table.contains (cell)) + { + setTextMode (null) + return + } + cell.dataset.countsText = '' + table.focus ({ preventScroll: true }) + const range = document.createRange () + range.setStart (nodes[0], 0) + const last = nodes[nodes.length - 1] + range.setEnd (last, last.textContent?.length ?? 0) + const text = window.getSelection () + text?.removeAllRanges () + text?.addRange (range) + return () => { + delete cell.dataset.countsText + const text = window.getSelection () + if (text != null && cell.contains (text.anchorNode)) text.removeAllRanges () + } + }, [identity, textMode]) + useLayoutEffect (() => { + if (textCell == null) return + const updateCaret = () => { + const text = window.getSelection () + if (!document.hasFocus () || text == null || !text.isCollapsed || text.rangeCount === 0 + || !textCell.contains (text.focusNode)) + { + setCaret (null) + return + } + const range = text.getRangeAt (0).cloneRange () + const rect = range.getBoundingClientRect () + const node = text.focusNode + const offset = text.focusOffset + const probe = range.cloneRange () + if (rect.height === 0 && node?.nodeType === Node.TEXT_NODE && node.textContent) + { + if (offset < node.textContent.length) probe.setEnd (node, offset + 1) + else probe.setStart (node, offset - 1) + } + const bounds = rect.height > 0 ? rect : probe.getBoundingClientRect () + if (bounds.height === 0) + { + setCaret (null) + return + } + const cell = textCell.getBoundingClientRect () + const left = (rect.height === 0 && offset === node?.textContent?.length) + ? bounds.right : bounds.left + setCaret ({ cell: textCell, + left: left - cell.left - textCell.clientLeft + textCell.scrollLeft, + top: bounds.top - cell.top - textCell.clientTop + textCell.scrollTop, + height: bounds.height }) + } + updateCaret () + document.addEventListener ('selectionchange', updateCaret) + document.addEventListener ('scroll', updateCaret, true) + window.addEventListener ('resize', updateCaret) + window.addEventListener ('focus', updateCaret) + window.addEventListener ('blur', updateCaret) + return () => { + document.removeEventListener ('selectionchange', updateCaret) + document.removeEventListener ('scroll', updateCaret, true) + window.removeEventListener ('resize', updateCaret) + window.removeEventListener ('focus', updateCaret) + window.removeEventListener ('blur', updateCaret) + } + }, [textCell]) + useEffect (() => { + if (textCell == null) return + const outside = (event: PointerEvent) => { + if (!textCell.contains (event.target as Node)) clearTextSelection () + } + document.addEventListener ('pointerdown', outside, true) + return () => document.removeEventListener ('pointerdown', outside, true) + }, [clearTextSelection, textCell]) + const bounds = useMemo (() => selection == null ? null : { + top: Math.min (selection.anchor.row, selection.end.row), + bottom: Math.max (selection.anchor.row, selection.end.row), + left: Math.min (selection.anchor.column, selection.end.column), + right: Math.max (selection.anchor.column, selection.end.column) }, [selection]) + const finish = () => { + mousePress.current = null + dragging.current = false + touchArmed.current = false + if (touchPress.current != null) clearTimeout (touchPress.current.timer) + touchPress.current = null + } + useEffect (() => { + const table = root.current + const preventPan = (event: TouchEvent) => { + if (touchArmed.current) event.preventDefault () + } + table?.addEventListener ('touchmove', preventPan, { passive: false }) + return () => { + table?.removeEventListener ('touchmove', preventPan) + if (touchPress.current != null) clearTimeout (touchPress.current.timer) + } + }, []) + useLayoutEffect (() => { + if (root.current == null) return + if (textCell != null && !root.current.contains (textCell)) setTextMode (null) + const grid = cellGrid (root.current) + const assigned = new Set () + for (const [row, cells] of grid.entries ()) + { + for (const [column, cell] of cells.entries ()) + { + if (assigned.has (cell)) continue + assigned.add (cell) + const selected = bounds != null && row <= bounds.bottom + && row + cell.rowSpan - 1 >= bounds.top && column <= bounds.right + && column + cell.colSpan - 1 >= bounds.left + cell.id = `${id}-${row}-${column}` + if (cell.getAttribute ('role') !== 'rowheader') cell.setAttribute ('role', 'gridcell') + cell.setAttribute ('aria-selected', String (selected)) + if (selected && cell !== textCell) cell.dataset.countsSelected = '' + else delete cell.dataset.countsSelected + } + } + if (bounds != null) + root.current.setAttribute ('aria-activedescendant', grid[bounds.bottom]?.[bounds.right]?.id ?? '') + else root.current.removeAttribute ('aria-activedescendant') + }, [bounds, children, id, textCell]) + const pointFor = (element: Element | null, y?: number): Point | null => { + const table = root.current + if (table == null || element == null || !table.contains (element)) return null + let cell = element.closest ('td') + while (cell != null && cell.closest ('table') !== table) + cell = cell.closest ('table')?.parentElement?.closest ('td') ?? null + if (cell == null || cell.parentElement?.parentElement !== table.tBodies[0]) return null + const grid = cellGrid (table) + const hitRow = y == null ? -1 : Array.from (table.tBodies[0].rows).findIndex (row => { + const rect = row.getBoundingClientRect () + return y >= rect.top && y < rect.bottom + }) + for (const [row, cells] of grid.entries ()) + { + if (hitRow >= 0 && row !== hitRow) continue + const column = cells.indexOf (cell as HTMLTableCellElement) + if (column >= 0) return { row, column } + } + return null + } + const enterTextSelection = (point: Point) => { + const table = root.current + const cell = table == null ? null : cellAt (table, point) + if (table == null || cell == null) return + // Structured cells retain their independent inner grid instead of selecting its UI. + const nested = cell.querySelector ('table') + if (nested != null) + { + nested.focus ({ preventScroll: true }) + return + } + const walker = document.createTreeWalker (cell, NodeFilter.SHOW_TEXT, { + acceptNode: node => { + const parent = node.parentElement + const auxiliaryFlg = parent?.closest ( + '[hidden],[aria-hidden="true"],.sr-only,svg,[data-counts-resize]') != null + return (parent?.closest ('table') === table && node.textContent?.trim () + && !auxiliaryFlg) + ? NodeFilter.FILTER_ACCEPT : NodeFilter.FILTER_REJECT + } }) + const nodes: Node[] = [] + for (let node = walker.nextNode (); node != null; node = walker.nextNode ()) + nodes.push (node) + if (nodes.length === 0) return + finish () + clearTextSelection () + setMenu (null) + setTextMode ({ identity, cell, nodes }) + } + const closeMenu = useCallback (() => { + setMenu (null) + root.current?.focus ({ preventScroll: true }) + }, []) + const openMenu = (point: Point, x: number, y: number) => { + const inside = bounds != null && point.row >= bounds.top && point.row <= bounds.bottom + && point.column >= bounds.left && point.column <= bounds.right + if (inside) setStored (selection) + else setStored ({ identity, anchor: point, end: point }) + setMenu ({ point, x, y, identity }) + } + const clipboard = (headers = false) => { + if (bounds == null || root.current == null) return null + if (copyTable != null) + { + const table = copyTable () + return countsClipboard ({ + columns: table.columns.slice (bounds.left, bounds.right + 1), + headerRows: table.headerRows?.map (row => row.slice (bounds.left, bounds.right + 1)), + rows: table.rows.slice (bounds.top, bounds.bottom + 1) + .map (row => row.slice (bounds.left, bounds.right + 1)) }, headers) + } + const rows: string[][] = [] + for (let row = bounds.top; row <= bounds.bottom; ++row) + { + const values: string[] = [] + for (let column = bounds.left; column <= bounds.right; ++column) + { + const cell = cellAt (root.current, { row, column }) + values.push (cell?.dataset.countsCopy ?? cell?.innerText ?? '') + } + rows.push (values) + } + if (headers) + { + const heading = root.current.tHead?.rows[0] + rows.unshift (Array.from ({ length: bounds.right - bounds.left + 1 }, (entry, index) => { + void entry + const cell = heading?.cells[bounds.left + index] + return cell?.dataset.countsHeading ?? cell?.textContent ?? '' + })) + } + return countsClipboard (rows, headers) + } + const menuAction = (action: CountsMenuAction) => { + if (menu == null || root.current == null) return + if (action === 'copy' || action === 'copyHeaders') + { + const payload = clipboard (action === 'copyHeaders') + if (payload != null) void writeCountsClipboard (payload).catch (error => { + console.error (error) + }) + } + else + { + const lastRow = (root.current.tBodies[0]?.rows.length ?? 1) - 1 + const lastColumn = columns.length - 1 + const anchor = { + row: action === 'row' ? menu.point.row : 0, + column: action === 'column' ? menu.point.column : 0 } + const end = { + row: action === 'row' ? menu.point.row : lastRow, + column: action === 'column' ? menu.point.column : lastColumn } + setStored ({ identity, anchor, end }) + } + closeMenu () + } + const copy = (event: ClipboardEvent) => { + const text = window.getSelection () + if (textCell != null && text != null && !text.isCollapsed + && textCell.contains (text.anchorNode) + && textCell.contains (text.focusNode)) + { + event.stopPropagation () + return + } + if (document.activeElement !== root.current) return + const payload = clipboard () + if (payload == null) return + event.preventDefault () + event.stopPropagation () + event.clipboardData.setData ('text/plain', payload.text) + try + { + event.clipboardData.setData ('text/html', payload.html) + } + catch + { + // Plain TSV remains available when the browser disallows HTML clipboard data. + } + } + const totalWidth = columns.reduce ((sum, key) => sum + layout.width (family, key), 0) + return ( + + tbody>tr>td[data-counts-selected]]:bg-blue-100', + '[&>tbody>tr>td[data-counts-selected]]:text-slate-900', + 'dark:[&>tbody>tr>td[data-counts-selected]]:bg-blue-900', + 'dark:[&>tbody>tr>td[data-counts-selected]]:text-slate-100', + '[&>tbody>tr>td[data-counts-selected]]:outline', + '[&>tbody>tr>td[data-counts-selected]]:outline-blue-500', + '[&>tbody>tr>td[data-counts-selected]]:outline-offset-[-1px]', className)} + onCopy={copy} + onDoubleClick={event => { + const element = event.target as Element + if (element.closest ('table') !== event.currentTarget + || element.closest (interactive) != null) return + if (textCell?.contains (element)) + { + event.stopPropagation () + return + } + const point = pointFor (element, event.clientY) + if (point == null) return + event.preventDefault () + event.stopPropagation () + enterTextSelection (point) + }} + onClickCapture={event => { + if ((event.target as Element).closest ('table') !== event.currentTarget + || !suppressClick.current) return + event.preventDefault () + event.stopPropagation () + suppressClick.current = false + }} + onPointerDown={event => { + const element = event.target as Element + if (element.closest ('table') !== event.currentTarget) return + suppressClick.current = false + if (textCell != null && textCell.contains (element)) + { + event.stopPropagation () + return + } + clearTextSelection () + if (element.closest (interactive) != null || event.button !== 0) return + if (element instanceof HTMLElement && element.hasAttribute ('data-counts-scroll')) + { + const rect = element.getBoundingClientRect () + if (event.clientX >= rect.left + element.clientLeft + element.clientWidth + || event.clientY >= rect.top + element.clientTop + element.clientHeight) return + } + const point = pointFor (element, event.clientY) + if (point == null) return + event.stopPropagation () + setMenu (null) + event.currentTarget.focus ({ preventScroll: true }) + setStored ({ identity, anchor: event.shiftKey && selection != null + ? selection.anchor : point, end: point }) + finish () + if (event.pointerType === 'mouse') + { + mousePress.current = { x: event.clientX, y: event.clientY } + } + else + { + const table = event.currentTarget + const pointerId = event.pointerId + touchPress.current = { x: event.clientX, y: event.clientY, + timer: setTimeout (() => { + suppressClick.current = true + openMenu (point, event.clientX, event.clientY) + dragging.current = true + touchArmed.current = true + table.setPointerCapture (pointerId) + }, 450) } + } + }} + onPointerMove={event => { + if (mousePress.current != null) + { + if (event.buttons === 0) finish () + else if (!dragging.current && Math.hypot ( + event.clientX - mousePress.current.x, event.clientY - mousePress.current.y) > 5) + { + dragging.current = true + suppressClick.current = true + event.currentTarget.setPointerCapture (event.pointerId) + } + } + if (touchPress.current != null && !touchArmed.current && Math.hypot ( + event.clientX - touchPress.current.x, event.clientY - touchPress.current.y) > 8) + finish () + if (!dragging.current || event.buttons === 0) return + event.preventDefault () + const point = pointFor (document.elementFromPoint (event.clientX, event.clientY), event.clientY) + if (point == null) return + if (touchArmed.current && touchPress.current != null + && Math.hypot (event.clientX - touchPress.current.x, + event.clientY - touchPress.current.y) <= 8) return + setMenu (null) + event.stopPropagation () + setStored (previous => previous?.identity !== identity ? previous : { + ...previous, end: point }) + }} + onPointerUp={finish} onPointerCancel={finish} + onLostPointerCapture={event => { + if (event.target === event.currentTarget) finish () + }} + onContextMenu={event => { + const element = event.target as Element + if (element.closest ('table') !== event.currentTarget) return + if (textCell != null && textCell.contains (element)) + { + event.stopPropagation () + return + } + const point = pointFor (element, event.clientY) + if (point == null || element.closest ('[data-counts-resize]') != null) return + event.preventDefault () + event.stopPropagation () + openMenu (point, event.clientX, event.clientY) + }} + onKeyDown={event => { + if (event.target !== event.currentTarget) return + if (textCell == null + && (event.key === 'ContextMenu' || (event.shiftKey && event.key === 'F10'))) + { + const point = selection?.end ?? { row: 0, column: 0 } + const rect = cellAt (event.currentTarget, point)?.getBoundingClientRect () + if (rect != null) + { + event.preventDefault () + openMenu (point, rect.left, rect.bottom) + } + return + } + if (event.key === 'Escape' || (event.key === 'Enter' && textCell != null)) + { + const textModeFlg = textCell != null + clearTextSelection () + if (textModeFlg) + { + window.getSelection ()?.removeAllRanges () + event.currentTarget.focus ({ preventScroll: true }) + } + else setStored (null) + event.preventDefault () + event.stopPropagation () + return + } + if (event.key === 'Enter' && textCell == null && selection != null) + { + event.preventDefault () + enterTextSelection (selection.end) + return + } + if (textCell != null) + { + const boundaryFlg = event.key === 'Home' || event.key === 'End' + if (!boundaryFlg + && !['ArrowUp', 'ArrowDown', 'ArrowLeft', 'ArrowRight'].includes (event.key)) return + const text = window.getSelection () + if (text == null) return + if (boundaryFlg && (event.ctrlKey || event.metaKey)) + { + const nodes = textMode?.nodes.filter (node => textCell.contains (node)) ?? [] + const edge = event.key === 'Home' ? nodes[0] : nodes[nodes.length - 1] + if (edge == null) return + event.preventDefault () + event.stopPropagation () + const end = event.key === 'Home' ? 0 : edge.textContent?.length ?? 0 + if (event.shiftKey && text.anchorNode != null && textCell.contains (text.anchorNode)) + text.setBaseAndExtent (text.anchorNode, text.anchorOffset, edge, end) + else text.setPosition (edge, end) + return + } + if (typeof text.modify !== 'function') return + event.preventDefault () + event.stopPropagation () + const anchor = text.anchorNode + const offset = text.anchorOffset + const backward = ['ArrowLeft', 'ArrowUp', 'Home'].includes (event.key) + const vertical = event.key === 'ArrowUp' || event.key === 'ArrowDown' + const direction = (vertical || boundaryFlg) ? (backward ? 'backward' : 'forward') + : (backward ? 'left' : 'right') + const grain = boundaryFlg ? 'lineboundary' : (vertical ? 'line' + : ((event.ctrlKey || event.metaKey) ? 'word' : 'character')) + text.modify (event.shiftKey ? 'extend' : 'move', direction, grain) + if (!textCell.contains (text.focusNode)) + { + const nodes = textMode?.nodes.filter (node => textCell.contains (node)) ?? [] + const edge = backward ? nodes[0] : nodes[nodes.length - 1] + if (edge != null) + { + const end = backward ? 0 : edge.textContent?.length ?? 0 + if (event.shiftKey && anchor != null && textCell.contains (anchor)) + text.setBaseAndExtent (anchor, offset, edge, end) + else text.setPosition (edge, end) + } + } + return + } + const boundaryFlg = event.key === 'Home' || event.key === 'End' + if (!boundaryFlg + && !['ArrowUp', 'ArrowDown', 'ArrowLeft', 'ArrowRight'].includes (event.key)) return + const rows = event.currentTarget.tBodies[0]?.rows + if (rows == null || rows.length === 0) return + event.preventDefault () + const previous = selection?.end ?? { row: 0, column: 0 } + const key = (selection == null && !boundaryFlg) ? '' : event.key + const first = event.key === 'Home' + const vertical = boundaryFlg && (event.ctrlKey || event.metaKey) + const point = { + row: vertical ? (first ? 0 : rows.length - 1) + : Math.max (0, Math.min (rows.length - 1, previous.row + + (key === 'ArrowDown' ? 1 : (key === 'ArrowUp' ? -1 : 0)))), + column: (boundaryFlg && !vertical) ? (first ? 0 : columns.length - 1) + : Math.max (0, Math.min (columns.length - 1, previous.column + + (key === 'ArrowRight' ? 1 : (key === 'ArrowLeft' ? -1 : 0)))) } + setStored ({ identity, anchor: event.shiftKey && selection != null + ? selection.anchor : point, end: point }) + scrollToCell (event.currentTarget, point, previous) + }}> + + {columns.map (key => )} + {children} +
+ {label ('selection.hint')} + {textCell != null && label ('selection.textMode')} +
+ {textCell != null && caret != null && caret.cell === textCell && createPortal ( + ) +} + + +export default CountsSelectableTable diff --git a/frontend/src/components/counts/CountsSortHeader.tsx b/frontend/src/components/counts/CountsSortHeader.tsx new file mode 100644 index 0000000..c9de6dd --- /dev/null +++ b/frontend/src/components/counts/CountsSortHeader.tsx @@ -0,0 +1,151 @@ +import { useContext, useEffect, useRef } from 'react' + +import SortableItem from '@/components/common/SortableItem' +import CountsResizeHandle from '@/components/counts/CountsResizeHandle' +import { Button } from '@/components/ui/button' +import { cn } from '@/lib/utils' +import { useUITranslation } from '@/lib/useUITranslation' +import { CountsTableScope, useCountsTableLayout } from '@/lib/countsTableLayout' + +import type { CountSort } from '@/lib/counts' + + +const CountsSortHeader = ( + { field, label, sorts, change, structured = false, rowSpan, colSpan, + resizable = true, dragId, totalKind, disabled = false }: { + field: string + label: { text: string; language: string } + sorts: CountSort[] + change: (key: string, additive: boolean) => void + structured?: boolean + rowSpan?: number; colSpan?: number; resizable?: boolean + dragId?: string; totalKind?: string; disabled?: boolean }, +) => { + const { message } = useUITranslation ('counts') + const family = useContext (CountsTableScope) + const layout = useCountsTableLayout () + const heading = useRef (null) + useEffect (() => { + if (!resizable || colSpan != null && colSpan > 1) return + const node = heading.current + const text = node?.querySelector ('[data-counts-label]') + const button = text?.closest ('button') + if (node == null || text == null || button == null) return + const canvas = document.createElement ('canvas') + const context = canvas.getContext ('2d') + if (context == null) return + const measure = () => { + context.font = getComputedStyle (text).font + const padding = (element: Element) => { + const style = getComputedStyle (element) + return parseFloat (style.paddingLeft) + parseFloat (style.paddingRight) + + parseFloat (style.borderLeftWidth) + parseFloat (style.borderRightWidth) + } + const handles = Array.from (node.children).filter (child => child !== button) + .reduce ((sum, child) => sum + Math.max ( + child.scrollWidth, child.getBoundingClientRect ().width), 0) + const resize = node.parentElement?.querySelector ('[data-counts-resize]') + const resizeWidth = resize?.getBoundingClientRect ().width ?? 0 + const indicator = button.querySelector ('[data-counts-sort-indicator]') + const sortWidth = indicator?.getBoundingClientRect ().width ?? 0 + const gap = parseFloat (getComputedStyle (button).columnGap) || 0 + layout.fit (family, field, context.measureText (label.text).width + + handles + padding (button) + padding (node.parentElement!) + + sortWidth + gap + resizeWidth) + } + measure () + let active = true + void document.fonts.ready.then (() => { if (active) measure () }) + return () => { active = false } + }, [family, field, label.text, dragId, resizable, colSpan, layout]) + const press = useRef<{ x: number; y: number + timer: ReturnType } | null> (null) + const suppressClick = useRef (false) + const cancel = () => { + if (press.current != null) clearTimeout (press.current.timer) + press.current = null + } + useEffect (() => () => { + if (press.current != null) clearTimeout (press.current.timer) + }, []) + const priority = sorts.findIndex (sort => sort.key === field) + const direction = sorts[priority]?.direction + const state = priority < 0 ? 'none' : (direction === 'asc' ? 'ascending' : 'descending') + const title = message (structured ? 'sort.structured' : 'sort.action') + const aria = message (priority < 0 ? 'sort.unsortedLabel' + : (sorts.length === 1 ? 'sort.singleLabel' : 'sort.label'), { + column: label.text, priority: priority + 1, + direction: message (`sort.${state}`).text }) + + const render = (shell?: Parameters[0]['renderShell']>>[0]) => ( + 1 && priority >= 0) ? 'other' : state}> +
+ {shell?.handle} + +
+ {resizable && } + ) + return dragId == null ? render () : { }} renderShell={render}>{null} +} + + +export default CountsSortHeader diff --git a/frontend/src/components/counts/CountsSummaryResults.test.tsx b/frontend/src/components/counts/CountsSummaryResults.test.tsx new file mode 100644 index 0000000..22e88d7 --- /dev/null +++ b/frontend/src/components/counts/CountsSummaryResults.test.tsx @@ -0,0 +1,46 @@ +import { fireEvent, screen } from '@testing-library/react' +import { describe, expect, it, vi } from 'vitest' + +import CountsSummaryResults from '@/components/counts/CountsSummaryResults' +import CountsTableLayout from '@/components/counts/CountsTableLayout' +import { CountsDefinitionContext } from '@/lib/countsDefinitions' +import { postDefinition } from '@/test/fixtures/counts' +import { renderWithProviders } from '@/test/render' + +import type { CountsResult } from '@/lib/counts' + +const result: CountsResult = { + count: 4, page: 1, limit: 100, groupCount: 1, outputCount: 1, + rows: [{ field: 'resource_id' }], cols: [], columnValuesEditable: false, + rowHeaders: [{ values: ['__counts_private__'], kind: 'data' }, + { values: [], kind: 'grand_total' }], + columnHeaders: [{ values: [], kind: 'grand_total' }], + cells: [{ values: ['__counts_private__'], count: 4 }], + pivotTotals: [{ row: [], column: [], count: 4 }], axisLabels: {} } + +describe ('Counts summary privacy and row numbering', () => { + it ('copies masked labels and omits private links and legacy numbering', () => { + const { container } = renderWithProviders ( + + + , { route: '/ja/counts/posts' }) + const label = screen.getByText ('********') + expect (label).toHaveAttribute ('title', expect.stringContaining ('集計件数には含まれます')) + expect (label).toHaveClass ('text-red-600', 'dark:text-red-400') + const row = label.closest ('tr')! + expect (row.querySelector ('a')).toBeNull () + expect (container.innerHTML).not.toContain ('__counts_private__') + expect (screen.queryByText ('No.')).not.toBeInTheDocument () + expect (row.querySelectorAll ('td')).toHaveLength (2) + const table = screen.getByRole ('grid') + table.focus () + fireEvent.keyDown (table, { key: 'Home' }) + const setData = vi.fn () + fireEvent.copy (table, { clipboardData: { setData } }) + const plain = setData.mock.calls.find (entry => entry[0] === 'text/plain')?.[1] + expect (plain).toContain ('********') + expect (plain).not.toContain ('__counts_private__') + }) +}) diff --git a/frontend/src/components/counts/CountsSummaryResults.tsx b/frontend/src/components/counts/CountsSummaryResults.tsx new file mode 100644 index 0000000..53660a3 --- /dev/null +++ b/frontend/src/components/counts/CountsSummaryResults.tsx @@ -0,0 +1,318 @@ +import SortableColumns from '@/components/common/SortableColumns' +import { countsSummaryValue } from '@/lib/countsDisplay' +import PrefetchLink from '@/components/PrefetchLink' +import CountsSelectableTable from '@/components/counts/CountsSelectableTable' +import CountsSortHeader from '@/components/counts/CountsSortHeader' +import { changeCountSort, countsMaskedValue, countsRequestParams, countsResourcePath, + readCountSort, writeCountSort } + from '@/lib/counts' +import { countsAxisLabel, useCountsDefinition } + from '@/lib/countsDefinitions' +import { useLocale } from '@/lib/useLocale' +import { useUITranslation } from '@/lib/useUITranslation' + +import { queryRecordParams, setQueryRecords, readCountState, encodeCountState, setCountState } + from '@/lib/queryState' + +import type { CountsPivotHeader, CountsResource, CountsResult } from '@/lib/counts' +import type { CountsAxis } from '@/lib/countsDefinitions' + + +const CountsSummaryTable = ( + { resource, data, params, pending, update, dragIds = {} }: { + dragIds?: Record + resource: CountsResource; data: CountsResult; params: URLSearchParams + pending: boolean + update: (key: string, value: string, extra?: Record) => void }, +) => { + const definition = useCountsDefinition () + const locale = useLocale () + const { message } = useUITranslation ('counts') + const rows = data.rows ?? [] + const cols = data.cols ?? [] + const cells = data.cells ?? [] + const rowHeaders: CountsPivotHeader[] = data.rowHeaders ?? [...new Set ( + cells.map (cell => JSON.stringify (cell.values.slice (0, rows.length))))] + .map (key => ({ values: JSON.parse (key), kind: 'data' })) + const columnHeaders: CountsPivotHeader[] = data.columnHeaders + ?? (data.columnValues ?? [[]]).map (values => ({ values, kind: 'data' })) + const sorts = readCountSort (params.get ('sort')) + const timezone = params.get ('timezone') ?? 'UTC' + const change = (key: string, additive: boolean) => update ( + 'sort', writeCountSort (changeCountSort (sorts, key, additive))) + const display = (value: string | number | null, axis: CountsAxis, index: number) => + countsSummaryValue (value, axis, index, data, locale, timezone) + const aggregateLabel = (header: CountsPivotHeader) => message ( + header.labelKey ?? (header.kind === 'grand_total' ? (rows.length + cols.length === 1 + ? 'singleTotal' : 'grandTotal') : 'subtotal')).text + const leading = Math.max (rows.length, cols.length > 0 ? 1 : 0) + const levels = Math.max (1, cols.length) + const headingLevels = cols.length === 0 ? 1 : levels * 2 + const names = (header: CountsPivotHeader, axes: CountsAxis[], offset: number) => + header.values.map ((value, index) => display (value, axes[index], offset + index)) + const columnKey = (header: CountsPivotHeader) => + `${header.kind}:${JSON.stringify (header.values)}` + const measures = readCountState[]> ( + params, 'measures', []) + const measureKey = (header: CountsPivotHeader) => { + if (header.values.length === 0) return 'count' + const ids = [...new Set (measures.map (entry => String (entry.key)))] + const existing = ids.find (id => JSON.stringify (measures.filter (entry => entry.key === id) + .sort ((a, b) => Number (a.axis) - Number (b.axis)).map (entry => entry.value)) + === JSON.stringify (header.values)) + return existing ?? `m${Array.from ({ length: 100 }, (_, index) => index) + .find (index => !ids.includes (`m${index}`)) ?? 99}` + } + const changeMeasure = (header: CountsPivotHeader, additive: boolean) => { + const key = measureKey (header) + const next = new URLSearchParams (params) + const nextSort = changeCountSort (sorts, key, additive) + const active = new Set (nextSort.map (sort => sort.key)) + const records = measures.filter (record => + record.key !== key && active.has (String (record.key))) + setQueryRecords (next, 'measures', [...records, ...(active.has (key) + ? header.values.map ((value, axis) => ({ key, axis, value })) : [])]) + update ('sort', writeCountSort (nextSort), + queryRecordParams (next, 'measures')) + } + const keys = [ + ...Array.from ({ length: leading }, (_, index) => `a${index}`), + ...columnHeaders.map (columnKey)] + const full = new Map (cells.map (cell => [JSON.stringify (cell.values), cell.count])) + const totals = new Map ((data.pivotTotals ?? []).map (cell => [ + JSON.stringify ([cell.row, cell.column]), cell.count])) + const totalLabels = new Map ((data.pivotTotals ?? []).map (cell => [ + JSON.stringify ([cell.row, cell.column]), cell.labelKey])) + const count = (row: CountsPivotHeader, column: CountsPivotHeader) => + (row.values.length === rows.length && column.values.length === cols.length) + ? full.get (JSON.stringify ([...row.values, ...column.values])) ?? 0 + : totals.get (JSON.stringify ([row.values, column.values])) ?? 0 + const drill = (row: CountsPivotHeader, column: CountsPivotHeader) => { + const next = new URLSearchParams (params) + const existing = readCountState[]> ( + params, 'conditions', []) + setCountState (next, 'conditions', encodeCountState ('conditions', [...existing, + ...row.values.map ((value, index) => ({ ...rows[index], operator: 'axis_eq', value })), + ...column.values.map ((value, index) => ({ + ...cols[index], operator: 'axis_eq', value }))])) + next.set ('mode', 'details') + next.delete ('sort') + next.delete ('page') + return `${countsResourcePath (resource)}?${next}` + } + const rowLabels = rowHeaders.map (header => { + const labels = names (header, rows, 0) + if (header.kind !== 'data') labels.push (aggregateLabel (header)) + return [ + ...Array.from ({ length: leading }, (_, index) => labels[index] ?? '')] + }) + const spans = rowHeaders.map ((row, index) => rows.map ((_, axis) => { + if (row.values.length <= axis) return 1 + const prefix = JSON.stringify (row.values.slice (0, axis + 1)) + const previous = rowHeaders[index - 1] + if (previous != null && previous.values.length > axis + && JSON.stringify (previous.values.slice (0, axis + 1)) === prefix) return 0 + let span = 1 + while (index + span < rowHeaders.length) + { + const next = rowHeaders[index + span] + if (next.values.length <= axis + || JSON.stringify (next.values.slice (0, axis + 1)) !== prefix) break + ++span + } + return span + })) + rowLabels.forEach ((labels, row) => spans[row].forEach ((span, column) => { + if (span === 0) labels[column] = '' + })) + // This unmerged matrix is also the single source for TSV and HTML clipboard headings. + const valueHeadingRows = Array.from ({ length: levels }, (_, level) => [ + ...Array.from ({ length: leading }, (_, index) => + level === 0 && rows[index] != null + ? countsAxisLabel (rows[index], definition, locale).text : ''), + ...columnHeaders.map ((header, index) => { + if (cols.length === 0) return message ('count').text + if (level > header.values.length) return '' + if (level === header.values.length) + return header.kind === 'data' ? '' : aggregateLabel (header) + const prefix = header.values.slice (0, level + 1) + const previous = columnHeaders[index - 1] + if (level < levels - 1 && previous != null + && JSON.stringify (previous.values.slice (0, level + 1)) === JSON.stringify (prefix)) + return '' + return display (header.values[level], cols[level], rows.length + level) + })]) + const axisHeadingRows = cols.map ((axis, level) => [ + ...Array.from ({ length: leading }, () => ''), + ...columnHeaders.map ((header, index) => { + if (header.values.length < level) return '' + const previous = columnHeaders[index - 1] + const prefix = JSON.stringify (header.values.slice (0, level)) + return (previous != null && previous.values.length >= level + && JSON.stringify (previous.values.slice (0, level)) === prefix) + ? '' : countsAxisLabel (axis, definition, locale).text + })]) + const headingRows = cols.length === 0 ? valueHeadingRows + : axisHeadingRows.flatMap ((headings, level) => [headings, valueHeadingRows[level]]) + if (cols.length > 0) + { + for (let index = 0; index < leading; ++index) + { + headingRows[0][index] = valueHeadingRows[0][index] + headingRows[1][index] = '' + } + } + const axisHeaderCells = (level: number) => { + const output = [] + for (let index = 0; index < columnHeaders.length; ++index) + { + const header = columnHeaders[index] + if (header.values.length < level) continue + const prefix = JSON.stringify (header.values.slice (0, level)) + let span = 1 + while (index + span < columnHeaders.length) + { + const sibling = columnHeaders[index + span] + if (sibling.values.length < level + || JSON.stringify (sibling.values.slice (0, level)) !== prefix) break + ++span + } + const name = countsAxisLabel (cols[level], definition, locale) + output.push ({name.text}) + index += span - 1 + } + return output + } + const matrix = rowHeaders.map ((header, index) => [...rowLabels[index], + ...columnHeaders.map (column => String (count (header, column)))]) + const headerCells = (level: number) => { + const output = [] + for (let index = 0; index < columnHeaders.length; ++index) + { + const header = columnHeaders[index] + if (cols.length > 0 && level > header.values.length) continue + const aggregate = header.kind !== 'data' && level === header.values.length + const leaf = level === levels - 1 || aggregate + const text = + cols.length === 0 + ? message ('count').text + : (aggregate ? aggregateLabel (header) + : display (header.values[level], cols[level], rows.length + level)) + let span = 1 + if (!leaf) + { + const prefix = JSON.stringify (header.values.slice (0, level + 1)) + while (index + span < columnHeaders.length) + { + const sibling = columnHeaders[index + span] + if (sibling.values.length <= level + || JSON.stringify (sibling.values.slice (0, level + 1)) !== prefix) break + ++span + } + } + const sortKey = leaf ? measureKey (header) : `a${rows.length + level}` + const field = leaf ? columnKey (header) : sortKey + output.push ( 0) ? (levels - level) * 2 - 1 : 1} + totalKind={header.kind !== 'data' ? header.kind : undefined} + dragId={level === 0 && header.values.length > 0 + ? dragIds[JSON.stringify (header.values.slice (0, 1))] : undefined} + label={{ text, language: message ('count').language }} + sorts={sorts.map (sort => sort.key === sortKey ? { ...sort, key: field } : sort)} + change={(_, additive) => leaf ? changeMeasure (header, additive) + : change (sortKey, additive)}/>) + index += span - 1 + } + return output + } + return
+ ({ columns: keys.map ((_, index) => ({ label: headingRows[0][index] })), + headerRows: headingRows, rows: matrix })} + identity={`${countsRequestParams (params)}:${JSON.stringify (keys)}`} + className="counts-summary text-left text-sm"> + + {headingRows.map ((_, tier) => + {tier === 0 && <> + {Array.from ({ length: leading }, (_, index) => rows[index] == null + ? + : )} + } + {(cols.length > 0 && tier % 2 === 0) + ? axisHeaderCells (tier / 2) : headerCells (Math.floor (tier / 2))} + )} + + {rowHeaders.map ((row, index) => + {rowLabels[index].map ((value, offset) => { + const axis = offset + if (axis >= 0 && row.kind !== 'data' && axis > row.values.length) return null + const aggregate = axis >= 0 && row.kind !== 'data' && axis === row.values.length + const span = axis < 0 ? 1 : spans[index][axis] ?? 1 + return span === 0 ? null : = 0 ? 'rowheader' : undefined} data-counts-copy={value} + data-counts-aggregate={row.kind !== 'data' ? '' : undefined} + data-counts-total={row.kind !== 'data' ? row.kind : undefined} + data-counts-row-total={row.kind !== 'data' ? row.kind : undefined} + className="overflow-hidden break-words border px-2 py-1 dark:border-slate-700"> + {value === '********' ? {value} : value} + })} + {columnHeaders.map (column => + {[...row.values, ...column.values].includes (countsMaskedValue) + ? {count (row, column)} + : + {count (row, column)}} + )} + )} + +
+} + + +const CountsSummaryResults = (props: Parameters[0]) => { + const locale = useLocale () + const { message } = useUITranslation ('counts') + const headers = props.data.columnHeaders ?? [] + const columns = props.data.cols ?? [] + if (columns.length === 0 || props.data.columnValuesEditable === false) + return + const groups = [...new Set (headers.filter (header => header.values.length > 0) + .map (header => JSON.stringify (header.values.slice (0, 1))))] + const ids = groups.map ((_, index) => `group${index}`) + const dragIds = Object.fromEntries (groups.map ((group, index) => [group, ids[index]])) + return ({ + text: countsSummaryValue (JSON.parse (groups[ids.indexOf (id)])[0], columns[0], + props.data.rows?.length ?? 0, props.data, locale, + props.params.get ('timezone') ?? 'UTC'), language: message ('count').language })} + onChange={order => props.update ('column_order', encodeCountState ('column_order', + order.map (id => ({ value: JSON.parse (groups[ids.indexOf (id)])[0] }))))}> + {order => { + const position = (header: CountsPivotHeader) => header.values.length === 0 ? order.length + : order.indexOf (dragIds[JSON.stringify (header.values.slice (0, 1))]) + const sorted = [...headers].sort ((a, b) => position (a) - position (b)) + return + }} + +} + +export default CountsSummaryResults diff --git a/frontend/src/components/counts/CountsTable.test.tsx b/frontend/src/components/counts/CountsTable.test.tsx new file mode 100644 index 0000000..f3f8b9c --- /dev/null +++ b/frontend/src/components/counts/CountsTable.test.tsx @@ -0,0 +1,106 @@ +import { fireEvent, screen } from '@testing-library/react' +import { describe, expect, it, vi } from 'vitest' + +import CountsNestedTable from '@/components/counts/CountsNestedTable' +import CountsSelectableTable from '@/components/counts/CountsSelectableTable' +import CountsTableLayout from '@/components/counts/CountsTableLayout' +import { useCountsTableLayout } from '@/lib/countsTableLayout' +import { renderWithProviders } from '@/test/render' + +import type { CountField } from '@/lib/counts' + + +const renderTable = () => renderWithProviders ( + + + first cellsecond + third + + +, { route: '/ja/counts/posts' }) + + +describe ('Counts cell keyboard contract', () => { + it ('moves Home/End on rows and Ctrl+Home/End on columns with a stable Shift anchor', () => { + renderTable () + const table = screen.getByRole ('grid', { name: 'Grid' }) + table.focus () + fireEvent.keyDown (table, { key: 'End' }) + expect (screen.getByText ('second')).toHaveAttribute ('data-counts-selected') + fireEvent.keyDown (table, { key: 'End', ctrlKey: true, shiftKey: true }) + expect (screen.getByText ('second')).toHaveAttribute ('data-counts-selected') + expect (screen.getByRole ('textbox').closest ('td')).toHaveAttribute ('data-counts-selected') + expect (screen.getByText ('third')).not.toHaveAttribute ('data-counts-selected') + fireEvent.keyDown (table, { key: 'Home' }) + expect (screen.getByText ('third')).toHaveAttribute ('data-counts-selected') + fireEvent.keyDown (table, { key: 'Home', ctrlKey: true }) + expect (screen.getByText ('first').closest ('td')).toHaveAttribute ('data-counts-selected') + }) + + it.each (['Enter', 'Escape']) ('leaves text mode with %s and restores the cell focus', key => { + renderTable () + const table = screen.getByRole ('grid', { name: 'Grid' }) + const cell = screen.getByText ('first').closest ('td')! + table.focus () + fireEvent.keyDown (table, { key: 'Home' }) + fireEvent.keyDown (table, { key: 'Enter' }) + expect (cell).toHaveAttribute ('data-counts-text') + expect (window.getSelection ()?.toString ()).toBe ('first cell') + const setData = vi.fn () + fireEvent.copy (table, { clipboardData: { setData } }) + expect (setData).not.toHaveBeenCalled () + fireEvent.keyDown (table, { key }) + expect (cell).not.toHaveAttribute ('data-counts-text') + expect (cell).toHaveAttribute ('data-counts-selected') + expect (window.getSelection ()?.toString ()).toBe ('') + expect (table).toHaveFocus () + fireEvent.copy (table, { clipboardData: { setData } }) + expect (setData).toHaveBeenCalledWith ('text/plain', 'first cell') + expect (setData).toHaveBeenCalledWith ('text/html', expect.stringContaining ('first cell')) + }) + + it ('does not consume input keyboard events and uses the sum of column widths', () => { + renderTable () + const input = screen.getByRole ('textbox') + input.focus () + expect (fireEvent.keyDown (input, { key: 'Enter' })).toBe (true) + expect (input).toHaveFocus () + expect (screen.getByRole ('grid', { name: 'Grid' })).toHaveStyle ({ width: '176px' }) + }) +}) + + +const WidthControls = () => { + const layout = useCountsTableLayout () + return <> + {layout.width ('details', 'title')} + + + +} + +it ('caps automatic header sizing and preserves manual widths on subsequent fitting', () => { + renderWithProviders () + fireEvent.click (screen.getByRole ('button', { name: 'fit' })) + expect (screen.getByRole ('status')).toHaveTextContent ('640') + fireEvent.click (screen.getByRole ('button', { name: 'resize' })) + fireEvent.click (screen.getByRole ('button', { name: 'fit' })) + expect (screen.getByRole ('status')).toHaveTextContent ('180') +}) + +it ('wraps nested JSON without truncating, reordering or changing its data', () => { + const fields: CountField[] = [ + { key: 'id', label: 'counts:columns.resourceId' }, + { key: 'name', label: 'counts:columns.name' }] + const rows = [{ id: 1, name: `https://example.invalid/${'a'.repeat (300)}` }, + { id: 2, name: 'line one\nline two "quoted"' }] + const params = new URLSearchParams ({ children: 'tags.id tags.name', nested_json: '1' }) + const { container } = renderWithProviders (, { route: '/ja/counts/posts' }) + const pre = container.querySelector ('pre')! + expect (pre.textContent).toBe (JSON.stringify (rows)) + expect (pre).toHaveClass ('whitespace-pre-wrap', '[overflow-wrap:anywhere]', 'font-sans') + expect (pre).not.toHaveClass ('truncate', 'whitespace-pre', 'select-none') +}) diff --git a/frontend/src/components/counts/CountsTableLayout.tsx b/frontend/src/components/counts/CountsTableLayout.tsx new file mode 100644 index 0000000..1a4dc08 --- /dev/null +++ b/frontend/src/components/counts/CountsTableLayout.tsx @@ -0,0 +1,34 @@ +import { useState } from 'react' + +import { CountsLayoutContext, defaultCountWidths, MAX_COUNT_WIDTH, MIN_COUNT_WIDTH, + MAX_INITIAL_COUNT_WIDTH } + from '@/lib/countsTableLayout' + +import type { ReactNode } from 'react' + + +const CountsTableLayout = ({ children }: { children: ReactNode }) => { + const [widths, setWidths] = useState> ({ }) + const [headings, setHeadings] = useState> ({ }) + const width = (family: string, key: string) => + widths[`${family}/${key}`] + ?? Math.max (defaultCountWidths[key] ?? 160, headings[`${family}/${key}`] ?? 0) + const fit = (family: string, key: string, value: number) => { + const next = Math.ceil (Math.min (MAX_INITIAL_COUNT_WIDTH, value)) + setHeadings (previous => previous[`${family}/${key}`] === next ? previous + : { ...previous, [`${family}/${key}`]: next }) + } + const custom = (family: string, key: string) => widths[`${family}/${key}`] != null + const resize = (family: string, key: string, value: number) => { + const next = Math.round (Math.max (MIN_COUNT_WIDTH, Math.min (MAX_COUNT_WIDTH, value))) + setWidths (previous => previous[`${family}/${key}`] === next ? previous + : { ...previous, [`${family}/${key}`]: next }) + } + return ( + + {children} + ) +} + + +export default CountsTableLayout diff --git a/frontend/src/components/counts/countsTable.css b/frontend/src/components/counts/countsTable.css new file mode 100644 index 0000000..9ac5b96 --- /dev/null +++ b/frontend/src/components/counts/countsTable.css @@ -0,0 +1,112 @@ +.counts-grid > tbody > tr > td { + @apply h-6; +} + +.counts-summary > tbody > tr > td, +.counts-summary > thead > tr > th { + padding: 0.5rem 0.75rem; +} + +.counts-grid > tbody > tr > td[data-counts-text] { + position: relative; + outline: 1px solid #64748b; + outline-offset: -1px; + user-select: text; + -webkit-user-select: text; + -webkit-touch-callout: default; + cursor: text; +} + +.dark .counts-grid > tbody > tr > td[data-counts-text] { + outline-color: #94a3b8; +} + +.counts-text-caret { + position: absolute; + width: 1px; + background-color: currentColor; + pointer-events: none; + user-select: none; + animation: counts-caret-blink 1s step-end infinite; +} + +@keyframes counts-caret-blink { + 0%, 49% { opacity: 1; } + 50%, 100% { opacity: 0; } +} + +@media (prefers-reduced-motion: reduce) { + .counts-text-caret { animation: none; } +} + +.counts-grid > tbody > tr:nth-child(odd) > td:not([data-counts-selected]) { + @apply bg-white text-slate-900; +} + +.counts-grid > tbody > tr:nth-child(even) > td:not([data-counts-selected]) { + @apply bg-slate-50 text-slate-900; +} + +.dark .counts-grid > tbody > tr:nth-child(odd) > td:not([data-counts-selected]) { + @apply bg-slate-950 text-slate-100; +} + +.dark .counts-grid > tbody > tr:nth-child(even) > td:not([data-counts-selected]) { + @apply bg-slate-900 text-slate-100; +} + +.counts-grid > thead { + @apply bg-slate-200 text-slate-950; +} + +.counts-grid > thead > tr > th { + @apply border-b border-r border-slate-300 bg-slate-200 text-slate-950; +} + +.dark .counts-grid > thead, +.dark .counts-grid > thead > tr > th { + @apply border-slate-600 bg-slate-800 text-slate-50; +} + +.counts-grid > tbody > tr > td { + @apply border-r border-slate-200 dark:border-slate-700; + -webkit-touch-callout: none; +} + +.counts-grid > tbody > tr > td:has(> .counts-nested) { + padding: 8px 12px; +} + +.counts-nested .counts-grid > thead > tr > th, +.counts-nested .counts-grid > tbody > tr > td { + padding: 8px 12px; +} + +.counts-nested { + @apply border-l-2 border-slate-300 dark:border-slate-600; +} + +/* Aggregate kind is semantic; only aggregate rows receive a background. */ +.counts-grid > tbody > tr > td[data-counts-total], +.counts-grid > thead > tr > th[data-counts-total] { + @apply border-l border-l-slate-400 font-semibold dark:border-l-slate-500; +} + +.counts-grid > tbody > tr > td[data-counts-row-total='subtotal']:not([data-counts-selected]) { + background-color: #e2e8f0; + color: #0f172a; + font-weight: 600; +} +.counts-grid > tbody > tr > td[data-counts-row-total='grand_total']:not([data-counts-selected]) { + background-color: #cbd5e1; + color: #0f172a; + font-weight: 700; +} +.dark .counts-grid > tbody > tr > td[data-counts-row-total='subtotal']:not([data-counts-selected]) { + background-color: #334155; + color: #f1f5f9; +} +.dark .counts-grid > tbody > tr > td[data-counts-row-total='grand_total']:not([data-counts-selected]) { + background-color: #475569; + color: #fff; +} diff --git a/frontend/src/components/dialogues/DialogueProvider.tsx b/frontend/src/components/dialogues/DialogueProvider.tsx index f8e966f..10834cc 100644 --- a/frontend/src/components/dialogues/DialogueProvider.tsx +++ b/frontend/src/components/dialogues/DialogueProvider.tsx @@ -8,6 +8,7 @@ import { Dialog, DialogHeader, DialogTitle } from '@/components/ui/dialog' import { DialogueContext } from '@/lib/dialogues/useDialogue' +import { cn } from '@/lib/utils' import { useUITranslation } from '@/lib/useUITranslation' import type { FC, ReactNode } from 'react' @@ -155,6 +156,22 @@ const DialogueProvider: FC = ({ children }) => { }, [closeRequest, pendingIds]) + const dismiss = async (request: DialogueRequest) => { + if (request.kind === 'form' && request.options.onCancel != null) + { + setPendingIds (current => [...current, request.id]) + try + { + if (await request.options.onCancel () === false) return + } + finally + { + setPendingIds (current => current.filter (id => id !== request.id)) + } + } + closeRequest (request.id, request.kind === 'confirm' ? false : null) + } + const active = queue[0] const startActions = active?.kind === 'form' @@ -178,18 +195,17 @@ const DialogueProvider: FC = ({ children }) => { nestedConfirm?.parentId === active.id || pendingIds.includes (active.id) if (!(open) && !(blocked)) - closeRequest (active.id, active.kind !== 'confirm' && null) + dismiss (active) }}> { if (nestedConfirm?.parentId === active.id || pendingIds.includes (active.id)) event.preventDefault () @@ -237,11 +253,14 @@ const DialogueProvider: FC = ({ children }) => { ))} -
+
button]:w-auto' : 'flex-col')}> + + + + + {title.text} +
+ {definition != null && update ('order', value)} + displayJSON={enabled => update ('nested_json', enabled ? '1' : '0')} + displayRowNumbers={enabled => update ('row_numbers', enabled ? '1' : '0')} + displayThumbnail={enabled => { + const columns = selectedCountColumns (resource, params, definition) + .filter (column => column !== 'thumbnail') + update ('columns', (enabled ? ['thumbnail', ...columns] : columns).join (' ')) + }} + execute={next => void execute (next)} + actions={
+ {(['csv', 'json'] as const).map (format => )} +
}/>} + + {exportError != null &&

+ {label ('errors.exportStart')}
+ {countsErrorMessage (exportError.failure, locale).text}

} +
+ {columnLimit != null ?
+

{label ('columnLimit.title')}

+

{label ('columnLimit.message', { max: columnLimit })}

+
: ((definitionError || errorFlg || runError) + &&

{failureMessage.text}

)} +
+
+ {mounted && definition != null && data != null && <> +

+ {label (resultMode === 'summary' ? 'outputRows' : 'outputTotal', { value: data.outputCount ?? '—' })} + {resultMode === 'details' && data.excludedCountScope === 'current_query' + && data.excludedCount != null && data.excludedCount > 0 && ( + + {label ('excludedRecords', { count: data.excludedCount })})} +

+ {resultMode === 'summary' ? : ( + )} + + } +
+ {waiting &&
+ + + {label ('shared:messages.loadingEllipsis')} +
} +
+ +
+ ) +} + + +const CountsResourcePage = ({ resource }: { resource: CountsResource }) => { + const locale = useLocale () + const { data, error, isError: errorFlg } = useQuery ({ + queryKey: countsKeys.definition (resource, locale), + queryFn: () => fetchCountsDefinition (resource, locale), staleTime: Infinity }) + return + + +} + + +export default CountsResourcePage diff --git a/frontend/src/pages/help/HelpChangesPage.test.tsx b/frontend/src/pages/help/HelpChangesPage.test.tsx index 30a08f9..acfa22e 100644 --- a/frontend/src/pages/help/HelpChangesPage.test.tsx +++ b/frontend/src/pages/help/HelpChangesPage.test.tsx @@ -1,10 +1,11 @@ import { QueryClient, QueryClientProvider } from '@tanstack/react-query' -import { render, screen } from '@testing-library/react' +import { act, render, screen, waitFor, within } from '@testing-library/react' import { HelmetProvider } from 'react-helmet-async' import { beforeEach, describe, expect, it, vi } from 'vitest' import { apiGet } from '@/lib/api' -import App from '@/App' +import { helpKeys } from '@/lib/queryKeys' +import App, { pageRoutes } from '@/App' import { helpFixture } from '@/test/helpFixtures' vi.hoisted (() => { @@ -22,17 +23,30 @@ vi.mock ('@/lib/useClientBehaviourSettings', () => ({ describe ('Help global edition history', () => { beforeEach (() => { + window.history.replaceState (null, '', '/ja/help/changes') helpFixture.reset () vi.mocked (apiGet).mockResolvedValue ({ locales: [{ code: 'ja', name: '日本語' }] }) }) it ('routes /help/changes to published editions in date and numeric order', async () => { + // Resolve the real lazy route before waiting for its asynchronously loaded data. + const route = pageRoutes (null, () => {}).find (route => route.path === 'help/changes')! + expect (route.handle?.preload).toBeTypeOf ('function') + await act (async () => { await route.handle!.preload! ('ja') }) const client = new QueryClient ({ defaultOptions: { queries: { retry: false } } }) const { container } = render ( ) - const links = await screen.findAllByRole ('link') + // Wait for the query started by the actual route, separately from DOM discovery. + await waitFor (() => expect (client.getQueryCache ().find ({ + queryKey: helpKeys.changes ('ja') })).toBeDefined ()) + const changes = client.getQueryCache ().find ({ queryKey: helpKeys.changes ('ja') }) + await act (async () => { await changes!.promise }) + expect (changes!.state.status).toBe ('success') + expect (changes!.state.data).toHaveLength (6) + const table = await screen.findByRole ('table') + const links = within (table).getAllByRole ('link') const editions = links.map (link => link.getAttribute ('href')) .filter (href => href?.startsWith ('/ja/help/') && href.includes ('edition=')) expect (editions).toEqual ([ diff --git a/frontend/src/pages/posts/PostDetailPage.test.tsx b/frontend/src/pages/posts/PostDetailPage.test.tsx index a23d19a..cabe112 100644 --- a/frontend/src/pages/posts/PostDetailPage.test.tsx +++ b/frontend/src/pages/posts/PostDetailPage.test.tsx @@ -29,11 +29,10 @@ vi.mock ('@/components/PostEditForm', () => ({ default: () =>
Post edit form
, })) vi.mock ('framer-motion', () => ({ - motion: { - div: ({ children }: { children?: ReactNode }) =>
{children}
, - main: ({ children }: { children?: ReactNode }) =>
{children}
, - }, -})) + useReducedMotion: () => false, + motion: { + div: ({ children }: { children?: ReactNode }) =>
{children}
, + main: ({ children }: { children?: ReactNode }) =>
{children}
} })) const renderPage = (user = buildUser ({ role: 'member' })) => renderWithProviders ( diff --git a/frontend/src/pages/posts/PostImportReviewPage.test.tsx b/frontend/src/pages/posts/PostImportReviewPage.test.tsx index 3a27996..8620a10 100644 --- a/frontend/src/pages/posts/PostImportReviewPage.test.tsx +++ b/frontend/src/pages/posts/PostImportReviewPage.test.tsx @@ -15,10 +15,11 @@ const api = vi.hoisted (() => ({ vi.mock ('@/lib/api', () => api) vi.mock ('framer-motion', () => ({ - AnimatePresence: ({ children }: { children?: ReactNode }) => <>{children}, - motion: { - div: ({ children }: { children?: ReactNode }) =>
{children}
, - main: ({ children }: { children?: ReactNode }) =>
{children}
} })) + AnimatePresence: ({ children }: { children?: ReactNode }) => <>{children}, + useReducedMotion: () => false, + motion: { + div: ({ children }: { children?: ReactNode }) =>
{children}
, + main: ({ children }: { children?: ReactNode }) =>
{children}
} })) const metadata = (url: string, title: string) => ({ url, diff --git a/frontend/src/pages/posts/PostListPage.tsx b/frontend/src/pages/posts/PostListPage.tsx index b5a0aa5..3b10b11 100644 --- a/frontend/src/pages/posts/PostListPage.tsx +++ b/frontend/src/pages/posts/PostListPage.tsx @@ -61,7 +61,6 @@ const PostListPage: FC = () => { const keys = { tags: tagsKey, match, page, limit, - ...(query.get ('tag_refs') && { tagReferences: query.get ('tag_refs')! }), url: '', title: '', originalCreatedFrom: '', originalCreatedTo: '', createdFrom: '', createdTo: '', updatedFrom: '', updatedTo: '', order } as const @@ -105,8 +104,7 @@ const PostListPage: FC = () => { updatePage (Math.min (page + 1, totalPages)) }, } - : { }), - }), [page, totalPages, location.search])) + : { }) }), [page, totalPages, location.search])) useLayoutEffect (() => { scroll (0, 0) diff --git a/frontend/src/pages/posts/PostSearchPage.tsx b/frontend/src/pages/posts/PostSearchPage.tsx index 68ac2e5..f2b1c78 100644 --- a/frontend/src/pages/posts/PostSearchPage.tsx +++ b/frontend/src/pages/posts/PostSearchPage.tsx @@ -126,7 +126,6 @@ const PostSearchPage: FC = () => { const keys: FetchPostsParams = { tags: qTags, match: qMatch, page, limit, - ...(query.get ('tag_refs') && { tagReferences: query.get ('tag_refs')! }), url: qURL, title: qTitle, originalCreatedFrom: qOriginalCreatedFrom, diff --git a/frontend/src/pages/tags/TagHistoryPage.tsx b/frontend/src/pages/tags/TagHistoryPage.tsx index 4687ef2..37b75e7 100644 --- a/frontend/src/pages/tags/TagHistoryPage.tsx +++ b/frontend/src/pages/tags/TagHistoryPage.tsx @@ -15,6 +15,7 @@ import { apiPut } from '@/lib/api' import { useDialogue } from '@/lib/dialogues/useDialogue' import { clientScrollBehaviour } from '@/lib/clientAnimation' import { localeLanguage } from '@/lib/locales' +import { languageUIMessage } from '@/lib/i18n' import { postsKeys, tagsKeys, wikiKeys } from '@/lib/queryKeys' import { fetchTagChanges } from '@/lib/tags' import { useClientBehaviourSettings } from '@/lib/useClientBehaviourSettings' @@ -150,7 +151,7 @@ const TagVersionRows = ({ change, children }: { children: (primary: ReactNode) => ReactNode }) => { const locale = useLocale () const language = localeLanguage (locale) - const { message, label } = useUITranslation ('history') + const { label } = useUITranslation ('history') const { i18n } = useTranslation ('history') const panelId = useId () const names = snapshotNames (change.names) @@ -210,15 +211,14 @@ const TagVersionRows = ({ change, children }: { )} {otherLanguages.map ((code, index) => { const entry = nameChanges.find (value => value.languageCode === code) - const key = `languageNames.${code}` - const known = i18n.exists (key, { ns: 'history', lng: locale }) + const language = languageUIMessage (code, locale, i18n) const next = entry != null ? entry.current : names[code] ?? null const prev = entry != null ? entry.prev : next return ( - - {known ? message (key).text : code} + + {language.text} diff --git a/frontend/src/pages/theatres/TheatreDetailPage.test.tsx b/frontend/src/pages/theatres/TheatreDetailPage.test.tsx index 3f5eae4..86d45b4 100644 --- a/frontend/src/pages/theatres/TheatreDetailPage.test.tsx +++ b/frontend/src/pages/theatres/TheatreDetailPage.test.tsx @@ -65,12 +65,11 @@ vi.mock ('@/components/PostEditForm', () => ({ default: () =>
Post edit form
, })) vi.mock ('framer-motion', () => ({ - motion: { - aside: ({ children }: { children?: ReactNode }) => , - div: ({ children }: { children?: ReactNode }) =>
{children}
, - main: ({ children }: { children?: ReactNode }) =>
{children}
, - }, -})) + useReducedMotion: () => false, + motion: { + aside: ({ children }: { children?: ReactNode }) => , + div: ({ children }: { children?: ReactNode }) =>
{children}
, + main: ({ children }: { children?: ReactNode }) =>
{children}
} })) const currentPost = buildPost ({ id: 10, diff --git a/frontend/src/test/fixtures/counts.ts b/frontend/src/test/fixtures/counts.ts new file mode 100644 index 0000000..5e8fe44 --- /dev/null +++ b/frontend/src/test/fixtures/counts.ts @@ -0,0 +1,49 @@ +import type { CountField, CountsResult } from '@/lib/counts' +import type { CountsDefinition } from '@/lib/countsDefinitions' + +// camelCase transport fixture for the definitions/query endpoints, not the legacy API. +export const nameColumns: CountField[] = [ + { key: 'name_ja', label: 'counts:localeName', labelValues: { language: '日本語' }, + nameLocale: { code: 'ja', languageCode: 'ja' } }, + { key: 'name_ain', label: 'counts:localeName', labelValues: { language: '北海道アイヌ語' }, + nameLocale: { code: 'ain', languageCode: 'ain' } }, + { key: 'aliases_ja', label: 'counts:localeAliases', labelValues: { language: '日本語' }, + nameLocale: { code: 'ja', languageCode: 'ja' }, multipleValues: true }, + { key: 'aliases_ain', label: 'counts:localeAliases', labelValues: { language: '北海道アイヌ語' }, + nameLocale: { code: 'ain', languageCode: 'ain' }, multipleValues: true }] + +export const postDefinition: CountsDefinition = { + fields: [ + { key: 'resource_id', type: 'integer', nullable: false, operators: ['eq', 'ne'] }, + { key: 'version_no', type: 'integer', nullable: false, operators: ['eq'] }, + { key: 'title', type: 'string', nullable: true, operators: ['eq', 'like'] }, + { key: 'user', type: 'integer', nullable: true, operators: ['eq'] }, + { key: 'created_at', type: 'datetime', nullable: false, operators: ['eq', 'range'] }], + outputFields: [ + { key: 'resource_id', label: 'counts:columns.resourceId' }, + { key: 'version_no', label: 'counts:columns.version' }, + { key: 'latest', label: 'counts:columns.latest' }, + { key: 'title', label: 'counts:columns.postTitle' }, + { key: 'tags', label: 'counts:columns.tags', children: nameColumns }], + nameColumns, + columnAliases: [{ key: 'name_ain_kana', value: 'name_ain' }], + conditionAliases: [{ key: 'name_ain_kana', value: 'name_ain_kana' }], + axes: { latest: ['resource_id', 'created_at', 'user', 'tags'], + history: ['resource_id', 'created_at', 'user', 'tags', 'changed_field'] }, + grains: ['year', 'month', 'day'], + defaults: [ + { definition: 'latest', mode: 'summary', params: [ + { key: 'rows', value: '' }, { key: 'cols', value: '' }, + { key: 'conditions', value: '' }] }, + ...['latest', 'history'].map (definition => ({ definition, mode: 'details', params: [ + { key: 'columns', value: 'resource_id version_no latest title tags' }, + { key: 'children', value: 'tags.name_ja' }, + { key: 'sort', value: 'resource_id:asc version_no:asc' }] }))] } + +export const executionAt = '2024-03-01T00:00:00.000000Z' +export const detailResult: CountsResult = { + count: 1, page: 1, limit: 100, outputCount: 1, visibleOutputCount: 1, + excludedCount: 0, excludedCountScope: 'current_query', executionAt, + conditionLabels: {}, rows: [], cols: [], versions: [{ + id: 10, resourceId: 7, versionNo: 1, latest: true, createdAt: executionAt, + createdByUser: null, title: 'Recorded title', parents: [], tags: [] }] } diff --git a/frontend/src/test/setup.ts b/frontend/src/test/setup.ts index ef2725c..1f395af 100644 --- a/frontend/src/test/setup.ts +++ b/frontend/src/test/setup.ts @@ -1,6 +1,10 @@ import '@testing-library/jest-dom/vitest' import { cleanup } from '@testing-library/react' -import { afterEach, vi } from 'vitest' +import { afterEach, beforeEach, vi } from 'vitest' + +beforeEach (() => { + vi.spyOn (HTMLCanvasElement.prototype, 'getContext').mockReturnValue (null) +}) afterEach (() => { cleanup () @@ -16,10 +20,10 @@ window.scroll = function () { ; } -window.requestAnimationFrame = callback => { - callback (0) - return 0 -} +// Animation callbacks must advance time and must not recurse synchronously. +window.requestAnimationFrame = callback => window.setTimeout ( + () => callback (performance.now ()), 16) +window.cancelAnimationFrame = identifier => window.clearTimeout (identifier) Object.defineProperty (window, 'matchMedia', { configurable: true, diff --git a/frontend/src/types.ts b/frontend/src/types.ts index 86a3f8c..69cfa6f 100644 --- a/frontend/src/types.ts +++ b/frontend/src/types.ts @@ -20,7 +20,6 @@ export type FetchPostsParams = { url: string title: string tags: string - tagReferences?: string match: 'all' | 'any' originalCreatedFrom: string originalCreatedTo: string diff --git a/frontend/vitest.config.ts b/frontend/vitest.config.ts index 593ad38..d08c97d 100644 --- a/frontend/vitest.config.ts +++ b/frontend/vitest.config.ts @@ -7,6 +7,7 @@ export default mergeConfig ( viteConfig, defineConfig ({ test: { environment: 'jsdom', + testTimeout: 30_000, exclude: ['**/node_modules/**', '**/dist/**', 'server.ssr.test.mjs'], setupFiles: './src/test/setup.ts', globals: false } }))