コミットを比較
39
コミット
| 作成者 | SHA1 | 日付 | |
|---|---|---|---|
|
|
efbd640963 | ||
|
|
7fc2faeaf0 | ||
|
|
237331481f | ||
|
|
e857a33ec3 | ||
|
|
9f92dbfb06 | ||
|
|
549a45e9d5 | ||
|
|
7090f63ab6 | ||
|
|
7536330ecf | ||
|
|
ecd8b449e3 | ||
|
|
97e769248f | ||
|
|
d0d1738192 | ||
|
|
49721e6915 | ||
|
|
614a12e7aa | ||
|
|
cb3f4c600a | ||
|
|
8ea3d211dc | ||
|
|
d6cf992622 | ||
|
|
4f117c7ba3 | ||
|
|
f9f31812a9 | ||
|
|
2706789fa4 | ||
|
|
30b5fe5f05 | ||
|
|
53ea010f82 | ||
|
|
5493673996 | ||
|
|
1154babe45 | ||
|
|
2563b3fef3 | ||
|
|
d5bb9c8508 | ||
|
|
f44a436e20 | ||
|
|
c61d9d76e3 | ||
|
|
88711d2bd8 | ||
|
|
b4139f8370 | ||
|
|
f91f9d0eb5 | ||
|
|
3e8f209542 | ||
|
|
8642225b94 | ||
|
|
96a5bb8601 | ||
|
|
e71663bd22 | ||
|
|
52a8d62be5 | ||
|
|
3e05757820 | ||
|
|
d411b3b40f | ||
|
|
304090f717 | ||
|
|
084a090cab |
@@ -1,5 +1,60 @@
|
||||
# AGENTS.md
|
||||
|
||||
## Mandatory security and URL contracts
|
||||
|
||||
- URL クエリへの JSON 格納を禁止する。Base64・16進数等による迂回も禁止。
|
||||
単純な配列は半角スペース区切り、構造化条件は番号付きの個別キーで表す。
|
||||
未指定と明示的な空選択を区別し、自由入力を区切り文字で分割しない。
|
||||
Frontend / Backend で同じ文法・上限・検証を使い、新しい URL 状態の追加前に確認する。
|
||||
- 一度除票されたリソースの除票以前(除票版を含む)の履歴は admin 以外へ公開しない。
|
||||
恢復しても解除しない。すべての後続除票を確定版番号と正しい複合キーで判定する。
|
||||
本文・属性・ID・版番号・関連参照・旧値を API / SSR / キャッシュ / 出力から漏らさない。
|
||||
サーバーが確定した current_user の role のみを認可に使用する。
|
||||
秘匿版も安全な全体件数には含めるが、公開明細・内訳・検索件数とは分離する。
|
||||
Counts では、現在の条件に対する権限除外前の出力行数と権限除外レコード数の
|
||||
数値公開及びその差分による間接的な情報開示を許容する。これは件数だけの例外であり、
|
||||
秘匿された本文・属性・識別情報・集計ラベルを公開する例外ではない。
|
||||
それ以外の経路で秘匿属性を条件・軸・ソート・差分から推測できる実装は禁止する。
|
||||
|
||||
## Mandatory Counts data contract
|
||||
|
||||
- Counts の取得対象の ID・集合・件数・並び順を UI ロカールに依存させない。
|
||||
言語・文字体系を抽出条件に使う場合は明示的な項目・条件として扱う。
|
||||
- API は記録済みの全言語名と別名を保持する。履歴の名前を現在の名前で置換しない。
|
||||
名称・別名列は Language の言語コードから一言語一列で定義し、Locale ごとに増やさない。
|
||||
同じ言語の全文字体系を含め、script_code 等の記録属性を保持する。
|
||||
未登録値を別言語で補完しない。旧ロカール限定条件の互換処理では意味を無言で変えない。
|
||||
初期表示列だけは UI ロカールに合わせてよい。明示的な列指定・空選択を優先する。
|
||||
- 明細の内表は行集合を保持し、表示列・列順・列幅だけを表示設定として操作する。
|
||||
内表行を抽出・除外する表示機能を追加しない。内表を参照する抽出条件は外表の
|
||||
採否だけを決定し、一致した内表行だけを残す実装は禁止する。
|
||||
- 言語名はタグ一件の言語別列とし、言語名のために内表をもう一段増やさない。
|
||||
複数の別名は欠落させず、複数値の項目又は別名の内表として保持する。
|
||||
- JSON は対象タグの全言語登録値を保持する。CSV は選択された表示列を出力する。
|
||||
UI 翻訳済みラベルでデータをソート・識別したり、JSON の項目キーを変えたりしない。
|
||||
|
||||
## Mandatory naming contract
|
||||
|
||||
- メソッド・関数など、実際に処理を実行する callable の名前以外の自前識別子に
|
||||
動詞を使用しない。状態は名詞・形容詞・形容詞的な語で表す。今後の全開発に適用する。
|
||||
- 変数、引数、プロパティ、型定義内の項目、API・URL パラメータ、JSON キー、
|
||||
DB カラムにも適用する。`is_` / `is`、`has_` / `has`、`can_` / `can`、
|
||||
`should_` / `should` などの動詞・助動詞由来の接頭辞を使用しない。
|
||||
状態を格納する識別子に `show`、`enable` 等の動詞も使用しない。
|
||||
- 形容詞で状態を表せる場合はその語を使う。例:`is_latest` / `isLatest` は
|
||||
`latest`、`is_active` は `active`、`is_deleted` は `deleted`、
|
||||
`is_visible` は `visible`、表示状態は `visible`、有効状態は `enabled`。
|
||||
- 名詞で真偽値を表す場合は原則 `_flg`、camelCase では `Flg` を付ける。
|
||||
例:`is_admin` は `admin_flg`、`is_owner` は `owner_flg`、
|
||||
`isAdmin` は `adminFlg`。クエリパラメータでは意味が明確なら接尾辞を省略できる。
|
||||
- callable の `setState`、`onChange` 等や、外部ライブラリ・フレームワーク所定の
|
||||
API 名は変更しない。外部値を自前の状態へ保持する場合は境界で適切な別名を付ける。
|
||||
言語固有の表記規約を維持し、予約語や曖昧さは規則を満たす明確な代替名で避ける。
|
||||
- 改名時は Frontend / Backend、URL、API、出力の名称を整合させる。
|
||||
旧 URL・API の互換性は入出力境界で扱い、旧命名を新しい内部コードへ伝播させない。
|
||||
DB の改名は既存データを保持する追加マイグレーションで扱い、適用済みのものを
|
||||
書き換えない。メソッド名・外部 API・文字列値等を機械的な一括置換で破壊しない。
|
||||
|
||||
## Project overview
|
||||
|
||||
BTRC Hub / タグ広場 is a split Rails API and React frontend repository.
|
||||
@@ -573,10 +628,10 @@ and layout reuse, follow `frontend/AGENTS.md`.
|
||||
- When new user-facing wording appears necessary, ask the user for the exact
|
||||
wording and placement before implementing it.
|
||||
- Do not invent replacement copy when removing unrequested wording.
|
||||
- Do not create, modify, or run tests unless the user explicitly asks for
|
||||
test work. When the user asks for tests, keep working within the permitted
|
||||
test-file scope and rerun them until they pass or the remaining failure is
|
||||
clearly blocked.
|
||||
- 明示的な個別指示がない限り、テストの追加・修正・削除・実行を一切行わない。
|
||||
fixture、mock、snapshot、テスト補助コードも対象とする。
|
||||
既存のテスト差分を勝手に修正・差戻ししない。
|
||||
この規約はディレクトリ固有の自動的なテスト実行推奨より優先する。
|
||||
- Test-only work includes adding, updating, deleting, reorganising, or fixing
|
||||
SyntaxError in tests. During test-only work, do not modify production code.
|
||||
- During test-only work, do not change production constants, behaviour, API
|
||||
@@ -631,6 +686,19 @@ and layout reuse, follow `frontend/AGENTS.md`.
|
||||
|
||||
## Frontend rules
|
||||
|
||||
- 原則として、ユーザ向け UI 文言を TSX / TypeScript に直接記述しない。
|
||||
新しい文言は既存の i18n システムで管理し、適切な既存キーを先に確認する。
|
||||
不必要な重複定義を避け、新規機能固有の文言は適切な namespace にまとめる。
|
||||
- 対応 locale の辞書を整合させる。未翻訳項目には既存の日本語 fallback と
|
||||
`_untranslated` 規約を使い、language metadata を維持する。
|
||||
翻訳を捏造せず、未翻訳文字列を正式な翻訳として登録しない。
|
||||
- UI 文言の追加・変更時はビルド、静的検査、実画面の確認等で、辞書と
|
||||
SSR / hydration の整合及び翻訳キーの非露出を検証する。
|
||||
i18n の検証を理由にテスト関連作業を開始しない。
|
||||
- API の内部識別子、パラメータ、ユーザ入力データ、外部 URL は機械的に
|
||||
翻訳しない。ボタン、項目名、CSV ヘッダ、エラー、空状態、ツールチップ、
|
||||
アクセシビリティ用ラベルも UI 文言として扱う。
|
||||
|
||||
- Use `frontend/src/lib/api.ts` for API calls so headers and camelCase conversion stay consistent.
|
||||
- Add or reuse TanStack Query keys through `frontend/src/lib/queryKeys.ts`;
|
||||
avoid ad hoc query key arrays.
|
||||
|
||||
@@ -1,5 +1,43 @@
|
||||
# backend/AGENTS.md
|
||||
|
||||
## Mandatory security and URL contracts
|
||||
|
||||
- URL クエリへの JSON 格納を禁止する。Base64・16進数等による迂回も禁止。
|
||||
単純な配列は半角スペース区切り、構造化条件は番号付きの個別キーで表す。
|
||||
未指定と明示的な空選択を区別し、自由入力を区切り文字で分割しない。
|
||||
Frontend / Backend で同じ文法・上限・検証を使い、新しい URL 状態の追加前に確認する。
|
||||
- 一度除票されたリソースの除票以前(除票版を含む)の履歴は admin 以外へ公開しない。
|
||||
恢復しても解除しない。すべての後続除票を確定版番号と正しい複合キーで判定する。
|
||||
本文・属性・ID・版番号・関連参照・旧値を API / SSR / キャッシュ / 出力から漏らさない。
|
||||
サーバーが確定した current_user の role のみを認可に使用する。
|
||||
秘匿版も安全な全体件数には含めるが、公開明細・内訳・検索件数とは分離する。
|
||||
Counts では、現在の条件に対する権限除外前の出力行数と権限除外レコード数の
|
||||
数値公開及びその差分による間接的な情報開示を許容する。これは件数だけの例外であり、
|
||||
秘匿された本文・属性・識別情報・集計ラベルを公開する例外ではない。
|
||||
それ以外の経路で秘匿属性を条件・軸・ソート・差分から推測できる実装は禁止する。
|
||||
- 履歴読み取りは HistoryAccess の共通認可を必須とする。書込用 relation を公開しない。
|
||||
export は生成時と取得時に認可し、権限変更・除票後に古いファイルを再利用しない。
|
||||
|
||||
## Mandatory Counts data contract
|
||||
|
||||
- Counts のロカール非依存・内表行集合保持はルート AGENTS.md の必須契約に従う。
|
||||
UI 言語を名前の WHERE・ORDER BY へ暗黙に使わない。名前検索は記録済みの
|
||||
全言語又は明示された言語・文字体系を使う。関連名にも HistoryAccess を適用する。
|
||||
JSON は全言語名を保持し、CSV の列選択を理由に元スナップショットを削らない。
|
||||
名称・別名列は Language から生成し、Locale の script_code で同じ言語の値を落とさない。
|
||||
旧ロカール列の条件互換は入力境界に限定し、旧キーの文字体系限定を維持する。
|
||||
|
||||
## Mandatory naming contract
|
||||
|
||||
- ルート `AGENTS.md` の必須命名規約を適用する。メソッド等の callable 以外の
|
||||
自前識別子に動詞を使わず、形容詞の状態名、名詞の真偽値には `_flg` を使用する。
|
||||
例:`is_latest` は `latest`、`is_admin` は `admin_flg`。
|
||||
- 引数、属性、SQL alias、API・URL パラメータ、JSON キー、DB カラムも対象とする。
|
||||
Rails 等が定義する外部 API 名や、処理を実行するメソッド名は変更しない。
|
||||
- 旧 API・URL の互換性は解析・シリアライズ境界で扱い、内部定義は新名に統一する。
|
||||
DB カラムの改名はデータを保持する追加マイグレーションで設計し、適用済みの
|
||||
マイグレーションを書き換えない。マイグレーションの実行には個別指示を要する。
|
||||
|
||||
## Scope
|
||||
|
||||
These rules apply to work under `backend/`.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
class ApplicationController < ActionController::API
|
||||
include RequestAccess
|
||||
include ActionController::Cookies
|
||||
include ActionController::RequestForgeryProtection
|
||||
|
||||
@@ -8,34 +9,46 @@ class ApplicationController < ActionController::API
|
||||
|
||||
rescue_from ActiveRecord::RecordInvalid, with: :render_record_invalid
|
||||
rescue_from ActiveRecord::RecordNotUnique, with: :render_record_not_unique
|
||||
rescue_from(ActiveRecord::RecordNotFound) { head :not_found }
|
||||
rescue_from(UnsupportedLocale) { head :not_found }
|
||||
|
||||
before_action :reject_banned_ip_address!
|
||||
before_action :authenticate_user
|
||||
before_action :reject_banned_user!
|
||||
before_action :validate_explicit_locale!
|
||||
before_action :enforce_query_contract!
|
||||
around_action :with_history_authority
|
||||
rescue_from(HistoryAccess::Forbidden) { head :not_found }
|
||||
protect_from_forgery with: :exception
|
||||
self.forgery_protection_origin_check = true
|
||||
rescue_from ActionController::InvalidAuthenticityToken, with: :reject_invalid_csrf
|
||||
after_action :prevent_authenticated_caching
|
||||
|
||||
def current_user = @current_user
|
||||
|
||||
private
|
||||
|
||||
def enforce_query_contract!
|
||||
pairs = URI.decode_www_form(request.query_string)
|
||||
return head :bad_request if pairs.length > 1000 || pairs.map(&:first).uniq.length != pairs.length
|
||||
return head :bad_request if request.query_parameters.values.any? { !_1.is_a?(String) }
|
||||
if (controller_name.start_with?('counts_') &&
|
||||
(request.query_parameters.keys & ['headers', 'labels']).any?)
|
||||
return head :bad_request
|
||||
end
|
||||
rescue ArgumentError
|
||||
head :bad_request
|
||||
end
|
||||
|
||||
def with_history_authority(&action)
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
HistoryAccess.set(user: current_user, &action)
|
||||
end
|
||||
|
||||
def reject_invalid_csrf
|
||||
response.headers['X-CSRF-Invalid'] = '1'
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
head :forbidden
|
||||
end
|
||||
|
||||
def authenticate_user
|
||||
code = session[:inheritance_code]
|
||||
return if code.blank?
|
||||
|
||||
@current_user = User.find_by(inheritance_code: code)
|
||||
end
|
||||
|
||||
def prevent_authenticated_caching
|
||||
response.headers['Cache-Control'] = 'private, no-store' if current_user
|
||||
end
|
||||
@@ -79,19 +92,6 @@ class ApplicationController < ActionController::API
|
||||
render_validation_error base: ['すでに存在してゐます.']
|
||||
end
|
||||
|
||||
def reject_banned_ip_address!
|
||||
ip_address = IpAddress.find_by(ip_address: IPAddr.new(request.remote_ip).hton)
|
||||
return unless ip_address&.banned?
|
||||
|
||||
head :forbidden
|
||||
end
|
||||
|
||||
def reject_banned_user!
|
||||
return unless current_user&.banned?
|
||||
|
||||
head :forbidden
|
||||
end
|
||||
|
||||
def render_validation_error record = nil, fields: { }, base: [], status: :unprocessable_entity
|
||||
errors = { }
|
||||
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
module CountsListing
|
||||
extend ActiveSupport::Concern
|
||||
|
||||
def index
|
||||
locale = resolve_locale!
|
||||
resource = is_a?(CountsPostsController) ? 'posts' : 'tags'
|
||||
query = CountsQuery.new(version_scope, resource, request.query_parameters.except('locale')
|
||||
.merge('locale' => locale.code))
|
||||
result = { population_count_scope: 'all_recorded_versions', population_count: HistoryAccess.population_count(query.relation.klass), count: query.count, page: query.page, limit: query.limit,
|
||||
at: request.query_parameters['at'].presence || Time.current.utc.iso8601(6) }
|
||||
if query.mode == 'summary'
|
||||
result.merge!(query.summary)
|
||||
else
|
||||
result[:versions] = CountsVersionRepr.many(
|
||||
query.details, resource:, language_code: locale.language_code, history_flg: true)
|
||||
end
|
||||
render json: result
|
||||
rescue ArgumentError, TypeError
|
||||
head :bad_request
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,52 @@
|
||||
# Permanent storage URLs must not bypass a resource's later discard boundary.
|
||||
module HistoryStorageAccess
|
||||
extend ActiveSupport::Concern
|
||||
|
||||
included do
|
||||
include RequestAccess
|
||||
prepend_before_action :authorise_history_storage!
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def authorise_history_storage!
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
return unless request.get? || request.head?
|
||||
|
||||
reject_banned_ip_address!
|
||||
return if performed?
|
||||
|
||||
authenticate_user
|
||||
reject_banned_user!
|
||||
return if performed?
|
||||
|
||||
signed = params[:signed_blob_id] || params[:signed_id]
|
||||
blob = if signed
|
||||
ActiveStorage::Blob.find_signed(signed)
|
||||
elsif params[:encoded_key]
|
||||
payload = ActiveStorage.verifier.verified(params[:encoded_key], purpose: :blob_key)
|
||||
key = payload && (payload[:key] || payload['key'])
|
||||
if key
|
||||
original = key.start_with?('variants/') ? key.split('/')[1] : key
|
||||
ActiveStorage::Blob.find_by(key: original)
|
||||
end
|
||||
end
|
||||
return head :not_found unless blob
|
||||
|
||||
allowed = HistoryAccess.set(user: current_user) { HistoryAccess.blob_visible?(blob) }
|
||||
head :not_found unless allowed
|
||||
rescue ActiveSupport::MessageVerifier::InvalidSignature
|
||||
head :not_found
|
||||
end
|
||||
|
||||
# Override the engine's public-forever caching before a streaming response commits.
|
||||
def http_cache_forever(**_options)
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
yield
|
||||
end
|
||||
|
||||
def expires_in(*arguments, **options)
|
||||
super
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,28 @@
|
||||
# Canonical read-side session and BAN boundary, shared by API and storage controllers.
|
||||
module RequestAccess
|
||||
extend ActiveSupport::Concern
|
||||
|
||||
def current_user = @current_user
|
||||
|
||||
private
|
||||
|
||||
def authenticate_user
|
||||
code = session[:inheritance_code]
|
||||
return if code.blank?
|
||||
|
||||
@current_user = User.find_by(inheritance_code: code)
|
||||
end
|
||||
|
||||
def reject_banned_ip_address!
|
||||
ip_address = IpAddress.find_by(ip_address: IPAddr.new(request.remote_ip).hton)
|
||||
return unless ip_address&.banned?
|
||||
|
||||
head :forbidden
|
||||
end
|
||||
|
||||
def reject_banned_user!
|
||||
return unless current_user&.banned?
|
||||
|
||||
head :forbidden
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,139 @@
|
||||
class CountsExportsController < ApplicationController
|
||||
wrap_parameters false
|
||||
|
||||
def create_job
|
||||
locale = resolve_locale!
|
||||
payload = request.request_parameters.merge(request.query_parameters)
|
||||
session[:counts_export_owner] ||= SecureRandom.hex(32)
|
||||
job = CountsExportJobs.start(payload, session[:counts_export_owner], locale.language_code)
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
render json: { job: }, status: :accepted
|
||||
rescue CountsExportJobs::WorkerUnavailable
|
||||
render json: { error: 'export_worker_unavailable' }, status: :service_unavailable
|
||||
rescue CountsExportDownload::Busy
|
||||
head :too_many_requests
|
||||
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
|
||||
head :bad_request
|
||||
end
|
||||
|
||||
def job_status
|
||||
job_state(cancel: false)
|
||||
end
|
||||
|
||||
def cancel_job
|
||||
job_state(cancel: true)
|
||||
end
|
||||
|
||||
def start_download
|
||||
result = CountsExportJobs.download_attempt(params[:token], session[:counts_export_owner])
|
||||
return head :not_found unless result
|
||||
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
render json: result
|
||||
end
|
||||
|
||||
def prepare
|
||||
locale = resolve_locale!
|
||||
payload = request.request_parameters.merge(request.query_parameters)
|
||||
exporter = CountsExporter.new(payload, language_code: locale.language_code)
|
||||
session[:counts_export_owner] ||= SecureRandom.hex(32)
|
||||
token = CountsExportDownload.prepare(exporter, session[:counts_export_owner])
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
render json: { token: }
|
||||
rescue CountsExportDownload::Busy
|
||||
head :too_many_requests
|
||||
rescue CountsExportDownload::TooLarge
|
||||
head 413
|
||||
rescue CountsExportDownload::TimedOut
|
||||
head :gateway_timeout
|
||||
rescue CountsExportDownload::StorageFull, Errno::ENOSPC
|
||||
head :insufficient_storage
|
||||
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
|
||||
head :bad_request
|
||||
end
|
||||
|
||||
def download
|
||||
if params[:job].present? || params[:attempt].present?
|
||||
return head :method_not_allowed unless request.get?
|
||||
|
||||
return confirmed_download
|
||||
end
|
||||
file = CountsExportDownload.find(params[:token], session[:counts_export_owner])
|
||||
return head :not_found unless file
|
||||
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
send_file file.fetch('path'),
|
||||
filename: "counts-#{ file.fetch('resource') }.#{ file.fetch('format') }",
|
||||
type: file['format'] == 'json' ? 'application/json' : 'text/csv',
|
||||
disposition: 'attachment'
|
||||
end
|
||||
|
||||
def confirmed_download
|
||||
owner = session[:counts_export_owner]
|
||||
arguments = { file_token: params[:token], attempt: params[:attempt] }
|
||||
return head :not_found unless arguments[:attempt].present? &&
|
||||
CountsExportJobs.download_attempt(params[:job], owner, **arguments)
|
||||
|
||||
file = CountsExportDownload.find(params[:token], owner)
|
||||
unless file
|
||||
CountsExportJobs.download_attempt(params[:job], owner, **arguments, status: 'unavailable')
|
||||
return head :not_found
|
||||
end
|
||||
# Open before committing the response, but never buffer the entire export.
|
||||
stream = File.open(file.fetch('path'), 'rb')
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
send_file_headers! filename: "counts-#{ file.fetch('resource') }.#{ file.fetch('format') }",
|
||||
type: file['format'] == 'json' ? 'application/json' : 'text/csv',
|
||||
disposition: 'attachment'
|
||||
job = params[:job]
|
||||
authority = HistoryAccess.user
|
||||
self.response_body = Enumerator.new do |output|
|
||||
begin
|
||||
first = true
|
||||
while (chunk = stream.read(64 * 1024))
|
||||
output << chunk
|
||||
if first
|
||||
HistoryAccess.set(user: authority) do
|
||||
CountsExportJobs.download_attempt(job, owner, **arguments, status: 'started')
|
||||
end
|
||||
first = false
|
||||
end
|
||||
end
|
||||
ensure
|
||||
stream.close
|
||||
end
|
||||
end
|
||||
rescue IOError, SystemCallError
|
||||
CountsExportJobs.download_attempt(params[:job], owner, **arguments, status: 'failed')
|
||||
head :service_unavailable
|
||||
end
|
||||
private :confirmed_download
|
||||
|
||||
def job_state(cancel:)
|
||||
result = CountsExportJobs.status(params[:token], session[:counts_export_owner], cancel:)
|
||||
return head :not_found unless result
|
||||
|
||||
response.headers['Cache-Control'] = 'private, no-store'
|
||||
render json: result
|
||||
end
|
||||
private :job_state
|
||||
|
||||
def index
|
||||
locale = resolve_locale!
|
||||
payload = request.request_parameters.except('authenticity_token')
|
||||
.merge(request.query_parameters)
|
||||
exporter = CountsExporter.new(payload,
|
||||
language_code: locale.language_code)
|
||||
session[:counts_export_owner] ||= SecureRandom.hex(32)
|
||||
token = CountsExportDownload.prepare(exporter, session[:counts_export_owner])
|
||||
file = CountsExportDownload.find(token, session[:counts_export_owner])
|
||||
return head :not_found unless file
|
||||
|
||||
send_file file.fetch('path'),
|
||||
filename: "counts-#{ exporter.resource }.#{ exporter.format }",
|
||||
type: exporter.format == 'json' ? 'application/json' : 'text/csv',
|
||||
disposition: 'attachment'
|
||||
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
|
||||
head :bad_request
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,3 @@
|
||||
class CountsPostsController < PostVersionsController
|
||||
include CountsListing
|
||||
end
|
||||
@@ -0,0 +1,10 @@
|
||||
class CountsStatesController < ApplicationController
|
||||
def index
|
||||
locale = resolve_locale!
|
||||
result = CountsStateQuery.new(request.query_parameters,
|
||||
language_code: locale.language_code).call
|
||||
render json: result
|
||||
rescue ArgumentError, TypeError
|
||||
head :bad_request
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,38 @@
|
||||
class CountsTablesController < ApplicationController
|
||||
def definition
|
||||
locale = resolve_locale!
|
||||
definition = CountsResourceDefinition.new(params[:resource])
|
||||
render json: { fields: definition.metadata,
|
||||
output_fields: definition.output_fields,
|
||||
filter_fields: definition.filter_metadata,
|
||||
axes: { latest: definition.axes(false), history: definition.axes(true) },
|
||||
name_columns: definition.name_columns,
|
||||
column_aliases: definition.column_aliases,
|
||||
condition_aliases: definition.condition_aliases,
|
||||
grains: CountsAxes::GRAINS,
|
||||
defaults: CountsDefaults.metadata(params[:resource], locale: locale.code) }
|
||||
rescue ArgumentError
|
||||
head :bad_request
|
||||
end
|
||||
|
||||
def index
|
||||
resolve_locale!
|
||||
render json: CountsTableQuery.new(request.query_parameters).call
|
||||
rescue CountsTableQuery::TooManyColumns
|
||||
render json: { error: 'too_many_columns', max_columns: CountsTableQuery::MAX_COLUMNS },
|
||||
status: :unprocessable_entity
|
||||
rescue ActiveRecord::StatementTimeout => error
|
||||
Rails.logger.warn("counts_query_timeout: #{ error.class }")
|
||||
render json: { error: 'counts_query_timeout' }, status: :service_unavailable
|
||||
rescue ArgumentError, TypeError, KeyError, JSON::ParserError
|
||||
head :bad_request
|
||||
end
|
||||
|
||||
def candidates
|
||||
resolve_locale!
|
||||
render json: CountsCandidates.new(request.query_parameters.merge(
|
||||
'resource' => request.path_parameters[:resource])).call
|
||||
rescue ArgumentError, TypeError, KeyError
|
||||
head :bad_request
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,3 @@
|
||||
class CountsTagsController < TagVersionsController
|
||||
include CountsListing
|
||||
end
|
||||
@@ -8,7 +8,7 @@ class MaterialVersionsController < ApplicationController
|
||||
|
||||
offset = (page - 1) * limit
|
||||
|
||||
q = MaterialVersion.includes(:created_by_user)
|
||||
q = HistoryAccess.visible(MaterialVersion.includes(:created_by_user))
|
||||
q = q.where(material_id: params[:material_id]) if params[:material_id].present?
|
||||
q = q.where(tag_name: params[:tag].to_s.strip) if params[:tag].present?
|
||||
q = q.where(event_type: params[:event_type]) if params[:event_type].present?
|
||||
@@ -20,7 +20,7 @@ class MaterialVersionsController < ApplicationController
|
||||
.limit(limit)
|
||||
.offset(offset)
|
||||
|
||||
render json: { versions: serialise_versions(versions), count: }
|
||||
render json: { versions: serialise_versions(versions), count:, population_count: HistoryAccess.population_count(MaterialVersion) }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
@@ -14,21 +14,7 @@ class PostVersionsController < ApplicationController
|
||||
|
||||
offset = (page - 1) * limit
|
||||
|
||||
q = PostVersion.joins(<<~SQL.squish)
|
||||
LEFT JOIN
|
||||
post_versions prev
|
||||
ON
|
||||
prev.post_id = post_versions.post_id
|
||||
AND prev.version_no = post_versions.version_no - 1
|
||||
SQL
|
||||
.select('post_versions.*',
|
||||
'prev.title AS prev_title',
|
||||
'prev.url AS prev_url',
|
||||
'prev.thumbnail_base AS prev_thumbnail_base',
|
||||
'prev.tags_json AS prev_tags_json',
|
||||
'prev.video_ms AS prev_video_ms',
|
||||
'prev.original_created_from AS prev_original_created_from',
|
||||
'prev.original_created_before AS prev_original_created_before')
|
||||
q = HistoryAccess.visible(version_scope)
|
||||
q = q.where('post_versions.post_id = ?', post_id) if post_id
|
||||
if external_tag_id || (tag_id && !(Tag.exists?(id: tag_id)))
|
||||
q = q.where('JSON_CONTAINS(post_versions.tags_json,' +
|
||||
@@ -46,11 +32,23 @@ class PostVersionsController < ApplicationController
|
||||
.limit(limit)
|
||||
.offset(offset)
|
||||
|
||||
render json: { versions: serialise_versions(versions, locale.language_code), count: }
|
||||
render json: { versions: serialise_versions(versions, locale.language_code), count:, population_count: HistoryAccess.population_count(PostVersion) }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def version_scope
|
||||
PostVersion.with_previous_version
|
||||
.select('post_versions.*',
|
||||
'prev.title AS prev_title',
|
||||
'prev.url AS prev_url',
|
||||
'prev.thumbnail_base AS prev_thumbnail_base',
|
||||
'prev.tags_json AS prev_tags_json',
|
||||
'prev.video_ms AS prev_video_ms',
|
||||
'prev.original_created_from AS prev_original_created_from',
|
||||
'prev.original_created_before AS prev_original_created_before')
|
||||
end
|
||||
|
||||
def serialise_versions rows, language_code
|
||||
rows = rows.to_a
|
||||
user_ids = rows.map(&:created_by_user_id).compact.uniq
|
||||
@@ -103,7 +101,7 @@ class PostVersionsController < ApplicationController
|
||||
}
|
||||
missing_ids = ids - names.keys
|
||||
|
||||
NicoTagVersion
|
||||
HistoryAccess.visible(NicoTagVersion.all)
|
||||
.where(tag_id: missing_ids)
|
||||
.order(:tag_id, version_no: :desc)
|
||||
.pluck(:tag_id, :name)
|
||||
|
||||
@@ -38,7 +38,7 @@ class PostsController < ApplicationController
|
||||
offset = (page - 1) * limit
|
||||
|
||||
pt_max_sql =
|
||||
PostVersion
|
||||
HistoryAccess.visible(PostVersion.all)
|
||||
.select('post_id, MAX(created_at) AS max_updated_at')
|
||||
.group('post_id')
|
||||
.to_sql
|
||||
@@ -344,6 +344,7 @@ class PostsController < ApplicationController
|
||||
unless force
|
||||
base_version = post.post_versions.find_by!(version_no: base_version_no)
|
||||
|
||||
HistoryAccess.require!(base_version)
|
||||
base_snapshot = post_snapshot_from_version(base_version)
|
||||
current_snapshot = post_snapshot_from_record(post, locale)
|
||||
end
|
||||
@@ -449,24 +450,8 @@ class PostsController < ApplicationController
|
||||
private
|
||||
|
||||
def tag_filter_references literals
|
||||
raw = params[:tag_refs]
|
||||
return nil if raw.blank?
|
||||
|
||||
references = JSON.parse(raw)
|
||||
valid = references.is_a?(Array) && references.all? do |reference|
|
||||
next false unless reference.is_a?(Hash) && reference['name'].is_a?(String)
|
||||
|
||||
ids = [reference['tag_id'], reference['external_tag_id']].compact
|
||||
((ids.any? || reference['missing'] == true) &&
|
||||
ids.all? { _1.is_a?(Integer) && _1.positive? } &&
|
||||
(!reference.key?('missing') || (reference['missing'] == true && ids.empty?)) &&
|
||||
(!reference.key?('negative') || [true, false].include?(reference['negative'])))
|
||||
end
|
||||
raise ActionController::BadRequest unless valid
|
||||
return nil unless references.map { _1['name'] } == literals
|
||||
|
||||
references
|
||||
rescue JSON::ParserError, TypeError
|
||||
PostTagSearch.parse_references(params.to_unsafe_h, literals)
|
||||
rescue ArgumentError, JSON::ParserError, TypeError
|
||||
raise ActionController::BadRequest
|
||||
end
|
||||
|
||||
@@ -482,17 +467,7 @@ class PostsController < ApplicationController
|
||||
|
||||
def filter_posts_by_tags locale, tag_names, match_type
|
||||
references = tag_filter_references(tag_names)
|
||||
literals = tag_names.each_with_index.map do |raw_name, index|
|
||||
name = raw_name.sub(/\Anot:/i, '')
|
||||
name = TagName.canonicalise(locale, name).first unless references
|
||||
reference = references&.[](index)
|
||||
negative = if reference&.key?('negative')
|
||||
reference['negative']
|
||||
else
|
||||
raw_name.downcase.start_with?('not:')
|
||||
end
|
||||
{ name:, negative:, reference: }
|
||||
end
|
||||
literals = PostTagSearch.literals(locale, tag_names.join(' '), references:)
|
||||
|
||||
return Post.all if literals.empty?
|
||||
|
||||
@@ -523,31 +498,10 @@ class PostsController < ApplicationController
|
||||
end
|
||||
|
||||
def tagged_post_ids_for locale, name, reference: nil
|
||||
if reference
|
||||
matches = Post.none
|
||||
if reference['tag_id']
|
||||
matches = matches.or(Post.where(id: PostTag.where(
|
||||
tag_id: reference['tag_id']).select(:post_id)))
|
||||
end
|
||||
if reference['external_tag_id']
|
||||
matches = matches.or(Post.where(id: PostExternalTag.where(
|
||||
external_tag_id: reference['external_tag_id']).select(:post_id)))
|
||||
end
|
||||
return matches.select(:id)
|
||||
end
|
||||
|
||||
posts_by_internal_tags =
|
||||
Post
|
||||
.joins(tags: :tag_names)
|
||||
.where(tag_names: { language_code: locale.language_code, name: })
|
||||
.select(:id)
|
||||
|
||||
posts_by_external_tags =
|
||||
Post
|
||||
.joins(:external_tags)
|
||||
.where("CONCAT(external_tags.platform, ':', external_tags.name) = ?", name)
|
||||
|
||||
(posts_by_internal_tags + posts_by_external_tags).uniq(&:id)
|
||||
internal, external = PostTagSearch.references(locale, { name:, reference: })
|
||||
Post.where(id: PostTag.where(tag_id: Array(internal)).select(:post_id))
|
||||
.or(Post.where(id: PostExternalTag.where(external_tag_id: Array(external))
|
||||
.select(:post_id))).select(:id)
|
||||
end
|
||||
|
||||
def sync_post_tags! post, desired_tags, sections
|
||||
|
||||
@@ -42,7 +42,7 @@ class SitemapEntriesController < ApplicationController
|
||||
when 'descriptions'
|
||||
TagDescription.where(language_code: locale.language_code, tag_id: named_tags)
|
||||
.where.not(body: [nil, ''])
|
||||
.where(TagDescriptionVersion
|
||||
.where(HistoryAccess.visible(TagDescriptionVersion.all)
|
||||
.where('tag_description_versions.tag_id = tag_descriptions.tag_id')
|
||||
.where('tag_description_versions.language_code = tag_descriptions.language_code')
|
||||
.where('tag_description_versions.version_no = tag_descriptions.version_no')
|
||||
@@ -50,6 +50,7 @@ class SitemapEntriesController < ApplicationController
|
||||
when 'tag_groups'
|
||||
TagGroup.joins(:versions)
|
||||
.where('tag_group_versions.version_no = tag_groups.version_no')
|
||||
.where(Arel.sql(HistoryAccess.sql(TagGroupVersion)))
|
||||
when 'theatres'
|
||||
Theatre.kept
|
||||
end
|
||||
|
||||
@@ -49,6 +49,7 @@ class TagDescriptionsController < ApplicationController
|
||||
private
|
||||
|
||||
def representation description
|
||||
HistoryAccess.require_current!(description, TagDescriptionVersion)
|
||||
description.as_json(only: [
|
||||
:tag_id,
|
||||
:language_code,
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
class TagGroupsController < ApplicationController
|
||||
def index
|
||||
locale = resolve_locale!
|
||||
groups = TagGroup.includes(:created_by_user, :updated_by_user).order(:id)
|
||||
groups = HistoryAccess.current(TagGroup.all, TagGroupVersion).includes(:created_by_user, :updated_by_user).order(:id)
|
||||
render json: TagGroupRepr.many(locale, groups.to_a)
|
||||
end
|
||||
|
||||
|
||||
@@ -10,17 +10,7 @@ class TagVersionsController < ApplicationController
|
||||
|
||||
offset = (page - 1) * limit
|
||||
|
||||
q = TagVersion.joins(<<~SQL.squish)
|
||||
LEFT JOIN
|
||||
tag_versions prev
|
||||
ON
|
||||
prev.tag_id = tag_versions.tag_id
|
||||
AND prev.version_no = tag_versions.version_no - 1
|
||||
SQL
|
||||
.select('tag_versions.*', 'prev.name AS prev_name', 'prev.category AS prev_category',
|
||||
'prev.deprecated_at AS prev_deprecated_at',
|
||||
'prev.aliases AS prev_aliases', 'prev.names AS prev_names',
|
||||
'prev.parent_tag_ids AS prev_parent_tag_ids')
|
||||
q = HistoryAccess.visible(version_scope)
|
||||
q = q.where('tag_versions.tag_id = ?', tag_id) if tag_id
|
||||
|
||||
count = q.except(:select, :order, :limit, :offset).count
|
||||
@@ -29,11 +19,26 @@ class TagVersionsController < ApplicationController
|
||||
.limit(limit)
|
||||
.offset(offset)
|
||||
|
||||
render json: { versions: serialise_versions(versions, locale.language_code), count: }
|
||||
render json: { versions: serialise_versions(versions, locale.language_code), count:, population_count: HistoryAccess.population_count(TagVersion) }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def version_scope
|
||||
TagVersion.joins(<<~SQL.squish)
|
||||
LEFT JOIN
|
||||
tag_versions prev
|
||||
ON
|
||||
prev.tag_id = tag_versions.tag_id
|
||||
AND prev.version_no = tag_versions.version_no - 1
|
||||
AND #{ HistoryAccess.sql(TagVersion, 'prev') }
|
||||
SQL
|
||||
.select('tag_versions.*', 'prev.name AS prev_name', 'prev.category AS prev_category',
|
||||
'prev.deprecated_at AS prev_deprecated_at',
|
||||
'prev.aliases AS prev_aliases', 'prev.names AS prev_names',
|
||||
'prev.parent_tag_ids AS prev_parent_tag_ids')
|
||||
end
|
||||
|
||||
def serialise_versions rows, language_code
|
||||
rows = rows.to_a
|
||||
tag_ids = rows.map(&:tag_id).uniq
|
||||
|
||||
@@ -8,12 +8,12 @@ class WikiPagesController < ApplicationController
|
||||
|
||||
title = params[:title].to_s.strip
|
||||
descriptions =
|
||||
TagDescription
|
||||
HistoryAccess.current(TagDescription.all, TagDescriptionVersion)
|
||||
.where(language_code: locale.language_code)
|
||||
.includes(tag: :tag_names)
|
||||
|
||||
descriptions = descriptions.order(updated_at: :desc, id: :asc)
|
||||
tag_groups = TagGroup.order(updated_at: :desc, id: :asc)
|
||||
tag_groups = HistoryAccess.current(TagGroup.all, TagGroupVersion).order(updated_at: :desc, id: :asc)
|
||||
if title.present?
|
||||
pattern = "%#{ ApplicationRecord.sanitize_sql_like(title) }%"
|
||||
descriptions =
|
||||
@@ -41,13 +41,13 @@ class WikiPagesController < ApplicationController
|
||||
return head :not_found unless language.is_a?(String)
|
||||
return head :not_found unless Language.exists?(code: language)
|
||||
|
||||
names = TagDescription.joins(tag: :tag_names)
|
||||
names = HistoryAccess.current(TagDescription.all, TagDescriptionVersion).joins(tag: :tag_names)
|
||||
.where(language_code: locale.language_code, tag_names: { language_code: language })
|
||||
.pluck('tag_descriptions.tag_id', 'tag_descriptions.id', 'tag_names.name')
|
||||
candidates = names.map do |tag_id, description_id, name|
|
||||
WikiPageRepr.autolink_tag(locale, tag_id:, description_id:, name:, language:)
|
||||
end
|
||||
candidates.concat(TagGroup.order(:id).pluck(:id, :name).map do |id, name|
|
||||
candidates.concat(HistoryAccess.current(TagGroup.all, TagGroupVersion).order(:id).pluck(:id, :name).map do |id, name|
|
||||
WikiPageRepr.autolink_group(locale, id:, name:, language:)
|
||||
end)
|
||||
render json: candidates
|
||||
@@ -286,12 +286,12 @@ class WikiPagesController < ApplicationController
|
||||
|
||||
tag_group = id && TagGroup.find_by(id:)
|
||||
if tag_group
|
||||
return render json: tag_group_changes(TagGroupVersion.where(tag_group_id: tag_group.id))
|
||||
return render json: tag_group_changes(HistoryAccess.visible(TagGroupVersion.all).where(tag_group_id: tag_group.id))
|
||||
end
|
||||
|
||||
json = []
|
||||
unless id
|
||||
json.concat(tag_group_changes(TagGroupVersion.all))
|
||||
json.concat(tag_group_changes(HistoryAccess.visible(TagGroupVersion.all)))
|
||||
json.concat(description_changes(locale,
|
||||
TagDescription.where(language_code: locale.language_code)))
|
||||
json = json.each_with_index.sort_by do |row, index|
|
||||
@@ -327,7 +327,7 @@ class WikiPagesController < ApplicationController
|
||||
|
||||
def render_group_representation representation, version, body: nil, status: :ok
|
||||
body ||= TagGroupMarkdownRenderer.render(representation)
|
||||
versions = TagGroupVersion.where(tag_group_id: version.tag_group_id)
|
||||
versions = HistoryAccess.visible(TagGroupVersion.all).where(tag_group_id: version.tag_group_id)
|
||||
pred = versions.find_by(version_no: version.version_no - 1)&.id
|
||||
succ = versions.find_by(version_no: version.version_no + 1)&.id
|
||||
updated_at = version.created_at
|
||||
@@ -346,11 +346,11 @@ class WikiPagesController < ApplicationController
|
||||
end
|
||||
|
||||
def current_group_version group
|
||||
TagGroupVersion.find_by(tag_group_id: group.id, version_no: group.version_no)
|
||||
HistoryAccess.visible(TagGroupVersion.all).find_by(tag_group_id: group.id, version_no: group.version_no)
|
||||
end
|
||||
|
||||
def group_version group, version_id
|
||||
TagGroupVersion.find_by(tag_group_id: group.id, id: version_id)
|
||||
HistoryAccess.visible(TagGroupVersion.all).find_by(tag_group_id: group.id, id: version_id)
|
||||
end
|
||||
|
||||
def wiki_diff from_body, to_body
|
||||
@@ -373,7 +373,7 @@ class WikiPagesController < ApplicationController
|
||||
def tag_group_changes versions
|
||||
rows = versions.includes(:created_by_user).order(id: :desc).limit(200).to_a
|
||||
groups = TagGroup.where(id: rows.map(&:tag_group_id).uniq).index_by(&:id)
|
||||
predecessors = TagGroupVersion.where(tag_group_id: groups.keys,
|
||||
predecessors = HistoryAccess.visible(TagGroupVersion.all).where(tag_group_id: groups.keys,
|
||||
version_no: rows.map { _1.version_no - 1 })
|
||||
.pluck(:tag_group_id, :version_no, :id)
|
||||
.to_h { |group_id, number, version_id| [[group_id, number], version_id] }
|
||||
@@ -406,18 +406,18 @@ class WikiPagesController < ApplicationController
|
||||
end
|
||||
|
||||
def description_versions description
|
||||
TagDescriptionVersion.where(tag_id: description.tag_id,
|
||||
HistoryAccess.visible(TagDescriptionVersion.all).where(tag_id: description.tag_id,
|
||||
language_code: description.language_code)
|
||||
end
|
||||
|
||||
def tag_description_by_title locale, title
|
||||
TagDescription.joins(tag: :tag_names)
|
||||
HistoryAccess.current(TagDescription.all, TagDescriptionVersion).joins(tag: :tag_names)
|
||||
.find_by(language_code: locale.language_code,
|
||||
tag_names: { language_code: locale.language_code, name: title })
|
||||
end
|
||||
|
||||
def description_changes locale, descriptions
|
||||
versions = TagDescriptionVersion.where(language_code: locale.language_code,
|
||||
versions = HistoryAccess.visible(TagDescriptionVersion.all).where(language_code: locale.language_code,
|
||||
tag_id: descriptions.select(:tag_id))
|
||||
.order(id: :desc).limit(200).to_a
|
||||
return [] if versions.empty?
|
||||
@@ -425,7 +425,7 @@ class WikiPagesController < ApplicationController
|
||||
by_tag = descriptions.where(tag_id: versions.map(&:tag_id))
|
||||
.includes(tag: :tag_names).index_by(&:tag_id)
|
||||
users = User.where(id: versions.map(&:created_by_user_id)).index_by(&:id)
|
||||
predecessors = TagDescriptionVersion.where(
|
||||
predecessors = HistoryAccess.visible(TagDescriptionVersion.all).where(
|
||||
language_code: locale.language_code, tag_id: by_tag.keys,
|
||||
version_no: versions.map { |version| version.version_no - 1 })
|
||||
.pluck(:tag_id, :version_no, :id)
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
require 'digest'
|
||||
|
||||
# Request/worker authority comes only from a server-resolved user. Default is public.
|
||||
class HistoryAccess < ActiveSupport::CurrentAttributes
|
||||
attribute :user
|
||||
class Forbidden < StandardError; end
|
||||
|
||||
KEYS = { 'post_versions' => ['post_id'], 'tag_versions' => ['tag_id'],
|
||||
'nico_tag_versions' => ['tag_id'],
|
||||
'tag_description_versions' => ['tag_id', 'language_code'],
|
||||
'tag_group_versions' => ['tag_group_id'], 'material_versions' => ['material_id'] }.freeze
|
||||
|
||||
def self.revision
|
||||
values = KEYS.keys.map do |table|
|
||||
table.classify.constantize.unscoped.where(event_type: 'discard').maximum(:id)
|
||||
end
|
||||
Digest::SHA256.hexdigest(values.to_json)
|
||||
end
|
||||
|
||||
def self.authority
|
||||
{ 'policy_revision' => revision, 'admin' => admin?, 'actor_id' => user&.id }
|
||||
end
|
||||
|
||||
def self.authorised_artifact?(metadata)
|
||||
return false unless metadata['policy_revision'].is_a?(String)
|
||||
if metadata['admin']
|
||||
admin? && user.id == metadata['actor_id']
|
||||
else
|
||||
metadata['policy_revision'] == revision
|
||||
end
|
||||
end
|
||||
|
||||
def self.admin? = user&.admin? == true
|
||||
|
||||
def self.sql(model, row = model.table_name, references: true)
|
||||
return 'TRUE' if admin?
|
||||
|
||||
table = model.table_name
|
||||
keys = KEYS.fetch(table)
|
||||
matches = keys.map { "boundary.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ')
|
||||
boundaries = "SELECT #{ keys.join(', ') }, MAX(version_no) AS version_no FROM #{ table } " +
|
||||
"WHERE event_type = 'discard' GROUP BY #{ keys.join(', ') }"
|
||||
clauses = ["NOT EXISTS (SELECT 1 FROM (#{ boundaries }) boundary WHERE #{ matches } " +
|
||||
"AND boundary.version_no >= #{ row }.version_no)"]
|
||||
clauses += reference_sql(table, row) if references
|
||||
clauses.map { "(#{ _1 })" }.join(' AND ')
|
||||
end
|
||||
|
||||
def self.discard_reference(table, key, expression)
|
||||
"NOT EXISTS (SELECT 1 FROM (#{ discarded_keys(table, key) }) blocked " +
|
||||
"WHERE #{ expression.call("blocked.#{ key }") })"
|
||||
end
|
||||
|
||||
def self.discarded_keys(table, key)
|
||||
"SELECT #{ key } FROM #{ table } WHERE event_type = 'discard' GROUP BY #{ key }"
|
||||
end
|
||||
|
||||
def self.json_reference(table, key, document)
|
||||
"NOT EXISTS (SELECT 1 FROM JSON_TABLE(COALESCE(#{ document }, JSON_ARRAY()), " +
|
||||
"'$[*]' COLUMNS (reference_id BIGINT PATH '$' NULL ON EMPTY)) reference_entry " +
|
||||
"JOIN (#{ discarded_keys(table, key) }) blocked " +
|
||||
"ON reference_entry.reference_id = blocked.#{ key })"
|
||||
end
|
||||
|
||||
def self.list_reference(table, key, document)
|
||||
# Preserve FIND_IN_SET's exact token semantics, including empty tokens.
|
||||
array = "CONCAT('[', REPLACE(JSON_QUOTE(REPLACE(COALESCE(#{ document }, ''), " +
|
||||
"' ', ',')), ',', '\",\"'), ']')"
|
||||
"NOT EXISTS (SELECT 1 FROM JSON_TABLE(#{ array }, '$[*]' " +
|
||||
"COLUMNS (reference_id VARCHAR(255) PATH '$')) reference_entry " +
|
||||
"JOIN (#{ discarded_keys(table, key) }) blocked " +
|
||||
"ON reference_entry.reference_id = CAST(blocked.#{ key } AS CHAR))"
|
||||
end
|
||||
|
||||
def self.reference_sql(table, row)
|
||||
case table
|
||||
when 'post_versions'
|
||||
[['tag_versions', 'tag_id', 'tag_id'],
|
||||
['nico_tag_versions', 'tag_id', 'external_tag_id']].map do |target, key, field|
|
||||
json_reference(target, key, "JSON_EXTRACT(#{ row }.tags_json, '$[*].#{ field }')")
|
||||
end + [list_reference('post_versions', 'post_id', "#{ row }.parent_post_ids")]
|
||||
when 'tag_versions'
|
||||
[list_reference('tag_versions', 'tag_id', "#{ row }.parent_tag_ids")]
|
||||
when 'tag_description_versions'
|
||||
[discard_reference('tag_versions', 'tag_id', ->(id) { "#{ row }.tag_id = #{ id }" })]
|
||||
when 'material_versions'
|
||||
[discard_reference('tag_versions', 'tag_id', ->(id) { "#{ row }.tag_id = #{ id }" }),
|
||||
discard_reference('material_versions', 'material_id', ->(id) {
|
||||
"#{ row }.parent_id = #{ id }"
|
||||
})]
|
||||
when 'tag_group_versions'
|
||||
[['tag_versions', 'tag_id'], ['tag_group_versions', 'tag_group_id']].map do |target, field|
|
||||
discard_reference(target, field, ->(id) {
|
||||
['members', 'placements'].map do |column|
|
||||
"JSON_CONTAINS(COALESCE(JSON_EXTRACT(#{ row }.#{ column }, " +
|
||||
"'$**.#{ field }'), JSON_ARRAY()), CAST(#{ id } AS CHAR))"
|
||||
end.join(' OR ')
|
||||
})
|
||||
end
|
||||
else []
|
||||
end
|
||||
end
|
||||
|
||||
def self.visible(scope, row = scope.klass.table_name)
|
||||
scope.where(Arel.sql(sql(scope.klass, row)))
|
||||
end
|
||||
|
||||
def self.current(scope, version_model)
|
||||
return scope if admin?
|
||||
|
||||
table = scope.klass.table_name
|
||||
keys = KEYS.fetch(version_model.table_name)
|
||||
matches = keys.map do |key|
|
||||
column = scope.klass.column_names.include?(key) ? key : 'id'
|
||||
"current_snapshot.#{ key } = #{ table }.#{ column }"
|
||||
end.join(' AND ')
|
||||
boundary = "#{ matches } AND current_snapshot.version_no >= #{ table }.version_no " +
|
||||
"AND current_snapshot.event_type = 'discard'"
|
||||
scope = scope.where(Arel.sql("NOT EXISTS (SELECT 1 FROM #{ version_model.table_name } " +
|
||||
"current_snapshot WHERE #{ boundary })"))
|
||||
condition = "#{ matches } AND current_snapshot.version_no = #{ table }.version_no"
|
||||
scope.where(Arel.sql("NOT EXISTS (SELECT 1 FROM #{ version_model.table_name } " +
|
||||
"current_snapshot WHERE #{ condition } AND NOT (#{ sql(version_model, 'current_snapshot') }))"))
|
||||
end
|
||||
|
||||
def self.require_current!(record, version_model)
|
||||
return if admin?
|
||||
raise Forbidden unless current(record.class.where(id: record.id), version_model).exists?
|
||||
end
|
||||
|
||||
def self.blob_visible?(blob, seen = [])
|
||||
return true if admin?
|
||||
return false if seen.include?(blob.id)
|
||||
seen = seen + [blob.id]
|
||||
private_versions = MaterialVersion.where(file_blob_id: blob.id)
|
||||
.where(Arel.sql("NOT (#{ sql(MaterialVersion) })"))
|
||||
return false if private_versions.exists?
|
||||
|
||||
blob.attachments.each do |attachment|
|
||||
model, key = { 'Post' => [PostVersion, 'post_id'],
|
||||
'Material' => [MaterialVersion, 'material_id'] }[attachment.record_type]
|
||||
if model && model.where(key => attachment.record_id, event_type: 'discard').exists?
|
||||
return false
|
||||
end
|
||||
if attachment.record_type == 'ActiveStorage::VariantRecord'
|
||||
original = ActiveStorage::VariantRecord.find_by(id: attachment.record_id)&.blob
|
||||
return false unless original && blob_visible?(original, seen)
|
||||
end
|
||||
end
|
||||
true
|
||||
end
|
||||
|
||||
def self.require_many!(records)
|
||||
return if admin? || records.empty?
|
||||
|
||||
ids = records.map(&:id).uniq
|
||||
raise Forbidden unless visible(records.first.class.where(id: ids)).count == ids.length
|
||||
end
|
||||
|
||||
def self.require!(record)
|
||||
return record if admin? || visible(record.class.where(id: record.id)).exists?
|
||||
|
||||
raise Forbidden
|
||||
end
|
||||
|
||||
# This number deliberately ignores time, identity and content filters to prevent count probes.
|
||||
def self.population_count(model) = model.unscoped.count
|
||||
|
||||
# Explicit whole-resource statistics; Counts pages do not run this aggregate.
|
||||
def self.population_counts(model)
|
||||
total, excluded = model.unscoped.optimizer_hints('MAX_EXECUTION_TIME(10000)').pick(
|
||||
Arel.sql('COUNT(*)'),
|
||||
Arel.sql("COALESCE(SUM(CASE WHEN #{ sql(model) } THEN 0 ELSE 1 END), 0)"))
|
||||
{ population_count: total.to_i, excluded_count: excluded.to_i,
|
||||
excluded_count_scope: 'all_recorded_versions' }
|
||||
end
|
||||
end
|
||||
@@ -1,6 +1,12 @@
|
||||
class PostVersion < ApplicationRecord
|
||||
include VersionRecord
|
||||
|
||||
scope :with_previous_version, -> {
|
||||
joins('LEFT JOIN post_versions prev ON prev.post_id = post_versions.post_id ' +
|
||||
'AND prev.version_no = post_versions.version_no - 1 AND ' +
|
||||
HistoryAccess.sql(PostVersion, 'prev'))
|
||||
}
|
||||
|
||||
belongs_to :post
|
||||
belongs_to :parent, class_name: 'Post', optional: true
|
||||
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
class TagDescriptionVersion < ApplicationRecord
|
||||
include VersionRecord
|
||||
|
||||
before_create { SharedResourceIdAllocator.assign!(self) }
|
||||
end
|
||||
|
||||
@@ -6,6 +6,8 @@ module VersionRecord
|
||||
discard: 'discard',
|
||||
restore: 'restore' }.freeze
|
||||
|
||||
def require_history_access! = HistoryAccess.require!(self)
|
||||
|
||||
def readonly? = persisted?
|
||||
|
||||
included do
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
# Counts uses recorded values only; history/diff endpoints retain their own contract.
|
||||
class CountsVersionRepr
|
||||
# language_code remains a compatibility argument; it never selects recorded values.
|
||||
def self.many(rows, resource:, language_code:, at: nil, inclusive: true, history_flg: false)
|
||||
new(rows.to_a, resource:, language_code:, at:, inclusive:, history_flg:).serialise
|
||||
end
|
||||
|
||||
def initialize(rows, resource:, language_code:, at:, inclusive:, history_flg:)
|
||||
@rows = rows
|
||||
HistoryAccess.require_many!(rows)
|
||||
@at = at
|
||||
@inclusive = inclusive
|
||||
@resource = resource
|
||||
@history_flg = history_flg
|
||||
@users = User.where(id: rows.map(&:created_by_user_id).compact).pluck(:id, :name).to_h
|
||||
@languages = master_names(Language)
|
||||
@scripts = master_names(Script)
|
||||
@name_columns = CountsResourceDefinition.new('tags').name_columns
|
||||
@tag_versions = {}
|
||||
if resource == 'posts'
|
||||
load_tag_versions
|
||||
requests = @rows.flat_map do |row|
|
||||
Array(json(row.tags_json)).filter_map do |entry|
|
||||
[entry['external_tag_id'], parent_point(row)] if entry['external_tag_id']
|
||||
end
|
||||
end.uniq
|
||||
@external_versions = load_related_versions(NicoTagVersion, requests)
|
||||
end
|
||||
load_parent_versions if resource == 'tags'
|
||||
end
|
||||
|
||||
def serialise
|
||||
@rows.map do |row|
|
||||
resource_id = row.public_send(@resource == 'posts' ? :post_id : :tag_id)
|
||||
common = { id: row.id, resource_id:,
|
||||
version_no: row.version_no, latest: row.attributes['counts_latest'].to_i == 1,
|
||||
created_at: row.created_at.iso8601,
|
||||
created_by_user: row.created_by_user_id &&
|
||||
{ id: row.created_by_user_id, name: @users[row.created_by_user_id] } }
|
||||
common.merge(@resource == 'posts' ? post(row) : tag(row))
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def master_names(model)
|
||||
model.select('code AS counts_code', :name).to_h do |record|
|
||||
[record.attributes['counts_code'], record.name]
|
||||
end
|
||||
end
|
||||
|
||||
def json(value)
|
||||
value.is_a?(String) ? JSON.parse(value) : value
|
||||
end
|
||||
|
||||
def load_tag_versions
|
||||
pairs = @rows.flat_map { Array(json(_1.tags_json)) }
|
||||
.filter_map { |entry| [entry['tag_id'], entry['version_no']] if entry['tag_id'] }
|
||||
.uniq
|
||||
pairs.each_slice(500) do |batch|
|
||||
placeholders = batch.map { '(?, ?)' }.join(', ')
|
||||
scope = HistoryAccess.visible(TagVersion.all)
|
||||
if @at
|
||||
scope = scope.where("created_at #{ @inclusive ? '<=' : '<' } ?", @at)
|
||||
end
|
||||
scope.where(["(tag_id, version_no) IN (#{ placeholders })", *batch.flatten])
|
||||
.select(:tag_id, :version_no, :name, :category, :names, :created_at).each do |version|
|
||||
@tag_versions[[version.tag_id, version.version_no]] = version
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def names(value)
|
||||
(json(value) || {}).sort.map do |language, entry|
|
||||
aliases = entry.fetch('aliases', []).map do |alias_entry|
|
||||
alias_entry.merge('script_name' => @scripts[alias_entry['script_code']])
|
||||
end
|
||||
{ language_code: language, language_name: @languages[language],
|
||||
script_code: entry['script_code'], script_name: @scripts[entry['script_code']],
|
||||
name: entry['name'], auto_generated: entry['auto_generated'], aliases: }
|
||||
end
|
||||
end
|
||||
|
||||
def locale_names(values)
|
||||
@name_columns.to_h do |field|
|
||||
locale = field.fetch(:name_locale)
|
||||
if field[:multiple_values]
|
||||
entries = values.select { _1[:language_code] == locale[:language_code] }
|
||||
aliases = entries.flat_map do |entry|
|
||||
entry[:aliases].map { _1.merge('language_code' => entry[:language_code]) }
|
||||
end
|
||||
next [field.fetch(:key).to_sym, aliases]
|
||||
end
|
||||
entry = values.find do |value|
|
||||
value[:language_code] == locale[:language_code]
|
||||
end
|
||||
[field.fetch(:key).to_sym, entry && entry[:name]]
|
||||
end
|
||||
end
|
||||
|
||||
def post(row)
|
||||
snapshots = Array(json(row.tags_json)).map do |entry|
|
||||
version = @tag_versions[[entry['tag_id'], entry['version_no']]]
|
||||
version = nil if version && version.created_at > parent_point(row)
|
||||
external = @external_versions[[entry['external_tag_id'], parent_point(row).iso8601(6)]]
|
||||
recorded_names = version ? names(version.names) : []
|
||||
aliases = recorded_names.flat_map do |value|
|
||||
value[:aliases].map { _1.merge('language_code' => value[:language_code],
|
||||
'language_name' => value[:language_name]) }
|
||||
end
|
||||
{ id: entry['tag_id'] || entry['external_tag_id'],
|
||||
kind: entry['tag_id'] ? 'internal' : 'external',
|
||||
tag_id: entry['tag_id'], external_tag_id: entry['external_tag_id'],
|
||||
version_no: entry['version_no'],
|
||||
name: entry['name'] || version&.name || external&.fetch('name'),
|
||||
category: entry['category'] || version&.category,
|
||||
sections: entry.fetch('sections', []), names: recorded_names, aliases: }
|
||||
.merge(locale_names(recorded_names))
|
||||
end
|
||||
{ title: row.title, url: row.url, thumbnail_base: row.thumbnail_base,
|
||||
tags: snapshots, parents: row.parent_post_ids.to_s.split.map { { id: _1.to_i } },
|
||||
video_ms: row.video_ms,
|
||||
original_created_from: row.original_created_from&.iso8601,
|
||||
original_created_before: row.original_created_before&.iso8601 }
|
||||
end
|
||||
|
||||
def parent_point(row)
|
||||
@history_flg ? [@at, row.created_at].compact.min : @at || row.created_at
|
||||
end
|
||||
|
||||
def load_parent_versions
|
||||
requests = @rows.flat_map do |row|
|
||||
row.parent_tag_ids.to_s.split.map { [_1.to_i, parent_point(row)] }
|
||||
end.uniq
|
||||
@parent_versions = load_related_versions(TagVersion, requests)
|
||||
end
|
||||
|
||||
def load_related_versions(model, requests)
|
||||
result = {}
|
||||
table = model.table_name
|
||||
comparison = @inclusive ? '<=' : '<'
|
||||
requests.each_slice(100) do |batch|
|
||||
queries = batch.map do |id, point|
|
||||
scope = HistoryAccess.visible(model.where(tag_id: id))
|
||||
.where("created_at #{ comparison } ?", point)
|
||||
.where("NOT EXISTS (SELECT 1 FROM #{ table } newer " +
|
||||
"WHERE newer.tag_id = #{ table }.tag_id AND newer.created_at #{ comparison } ? " +
|
||||
"AND newer.version_no > #{ table }.version_no)", point)
|
||||
quoted = model.connection.quote(point.iso8601(6))
|
||||
names = model == TagVersion ? 'names' : 'JSON_OBJECT() AS names'
|
||||
scope.select("tag_id, name, #{ names }, #{ quoted } AS reference_point").to_sql
|
||||
end
|
||||
next if queries.empty?
|
||||
|
||||
model.connection.select_all(queries.map { "(#{ _1 })" }.join(' UNION ALL ')).each do |value|
|
||||
result[[value['tag_id'], value['reference_point']]] = value
|
||||
end
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
def tag(row)
|
||||
recorded_names = names(row.names)
|
||||
aliases = recorded_names.flat_map do |entry|
|
||||
entry[:aliases].map {
|
||||
_1.merge('language_code' => entry[:language_code],
|
||||
'language_name' => entry[:language_name])
|
||||
}
|
||||
end
|
||||
{ name: row.name, names: recorded_names,
|
||||
category: row.category, aliases:,
|
||||
parents: row.parent_tag_ids.to_s.split.map do |id|
|
||||
value = @parent_versions[[id.to_i, parent_point(row).iso8601(6)]]
|
||||
snapshots = value && json(value['names'])
|
||||
recorded = names(snapshots)
|
||||
{ id: id.to_i, name: value&.fetch('name'), names: recorded }.merge(locale_names(recorded))
|
||||
end,
|
||||
deprecated: row.deprecated_at&.iso8601 }.merge(locale_names(recorded_names))
|
||||
end
|
||||
end
|
||||
@@ -44,6 +44,7 @@ module TagGroupRepr
|
||||
end
|
||||
|
||||
def one locale, group
|
||||
HistoryAccess.require_current!(group, TagGroupVersion)
|
||||
many(locale, [group]).first
|
||||
end
|
||||
|
||||
@@ -56,10 +57,11 @@ module TagGroupRepr
|
||||
end
|
||||
|
||||
def current_version group
|
||||
TagGroupVersion.find_by(tag_group_id: group.id, version_no: group.version_no)
|
||||
HistoryAccess.visible(TagGroupVersion.all).find_by(tag_group_id: group.id, version_no: group.version_no)
|
||||
end
|
||||
|
||||
def historical locale, version
|
||||
HistoryAccess.require!(version)
|
||||
members = Array(version.members)
|
||||
placements = Array(version.placements)
|
||||
tag_references = members.filter_map { |member| member['tag_id'] } +
|
||||
@@ -153,13 +155,13 @@ module TagGroupRepr
|
||||
end
|
||||
|
||||
def historical_tag_versions ids
|
||||
TagVersion.where(tag_id: ids.uniq).to_a.index_by do |version|
|
||||
HistoryAccess.visible(TagVersion.all).where(tag_id: ids.uniq).to_a.index_by do |version|
|
||||
[version.tag_id, version.version_no]
|
||||
end
|
||||
end
|
||||
|
||||
def historical_group_versions ids
|
||||
TagGroupVersion.where(tag_group_id: ids.uniq).to_a.index_by do |version|
|
||||
HistoryAccess.visible(TagGroupVersion.all).where(tag_group_id: ids.uniq).to_a.index_by do |version|
|
||||
[version.tag_group_id, version.version_no]
|
||||
end
|
||||
end
|
||||
|
||||
@@ -5,6 +5,7 @@ module WikiPageRepr
|
||||
module_function
|
||||
|
||||
def tag_description locale, description
|
||||
HistoryAccess.require_current!(description, TagDescriptionVersion)
|
||||
tag = description.tag
|
||||
|
||||
{ 'id' => description.id,
|
||||
|
||||
@@ -0,0 +1,115 @@
|
||||
class CountsAxes
|
||||
MAX_AXES = 8
|
||||
MASKED_VALUE = '__counts_private__'.freeze
|
||||
PUBLIC_FIELDS = ['created_at', 'event_type'].freeze
|
||||
class TooManyAxes < ArgumentError; end
|
||||
class PrivateCondition < ArgumentError; end
|
||||
|
||||
GRAINS = ['year', 'quarter', 'month', 'week', 'day', 'hour'].freeze
|
||||
|
||||
def self.time_expression(expression, grain, timezone)
|
||||
raise ArgumentError unless GRAINS.include?(grain)
|
||||
|
||||
connection = ApplicationRecord.connection
|
||||
# Use the installed TZInfo database, including DST, rather than MySQL timezone tables.
|
||||
first = Time.utc(1900)
|
||||
last = Time.utc(2101)
|
||||
offset = timezone.period_for_utc(first).utc_total_offset
|
||||
clauses = timezone.transitions_up_to(last, first).filter_map do |transition|
|
||||
next if transition.offset.utc_total_offset == offset
|
||||
|
||||
previous = offset
|
||||
offset = transition.offset.utc_total_offset
|
||||
"WHEN #{ expression } < #{ connection.quote(transition.at.to_time) } THEN #{ previous }"
|
||||
end
|
||||
seconds = clauses.empty? ? offset.to_s : "CASE #{ clauses.join(' ') } ELSE #{ offset } END"
|
||||
local = "DATE_ADD(#{ expression }, INTERVAL (#{ seconds }) SECOND)"
|
||||
case grain
|
||||
when 'year' then "DATE_FORMAT(#{ local }, '%Y')"
|
||||
when 'quarter' then "CONCAT(YEAR(#{ local }), '-Q', QUARTER(#{ local }))"
|
||||
when 'month' then "DATE_FORMAT(#{ local }, '%Y-%m')"
|
||||
when 'week' then "DATE_FORMAT(#{ local }, '%x-W%v')"
|
||||
when 'day' then "DATE_FORMAT(#{ local }, '%Y-%m-%d')"
|
||||
else "DATE_FORMAT(#{ local }, '%Y-%m-%d %H:00')"
|
||||
end
|
||||
end
|
||||
|
||||
attr_reader :rows, :columns, :all
|
||||
|
||||
def initialize(population)
|
||||
@population = population
|
||||
@definition = population.definition
|
||||
@rows = parse(population.params.fetch('rows', ''))
|
||||
@columns = parse(population.params.fetch('cols', ''))
|
||||
@all = rows + columns
|
||||
raise TooManyAxes if all.length > MAX_AXES
|
||||
raise ArgumentError if all.map { _1['field'] }.uniq.length != all.length
|
||||
end
|
||||
|
||||
def expressions
|
||||
@expressions ||= all.map do |axis|
|
||||
field = axis.fetch('field')
|
||||
expression = case field
|
||||
when 'tags'
|
||||
"CASE WHEN tag_entry.tag_id IS NOT NULL " +
|
||||
"THEN CONCAT('internal:', tag_entry.tag_id) " +
|
||||
"ELSE CONCAT('external:', tag_entry.external_tag_id) END"
|
||||
when 'changed_field' then 'changed.field_key'
|
||||
else
|
||||
expression = @definition.expression(field)
|
||||
@definition.fields[field] == :datetime ? self.class.time_expression(
|
||||
expression, axis.fetch('grain', 'month'), @population.timezone) : expression
|
||||
end
|
||||
if !HistoryAccess.admin? && !PUBLIC_FIELDS.include?(field)
|
||||
expression = "CASE WHEN #{ HistoryAccess.sql(@definition.model, 'v') } " +
|
||||
"THEN CAST((#{ expression }) AS CHAR) ELSE '#{ MASKED_VALUE }' END"
|
||||
end
|
||||
expression
|
||||
end
|
||||
end
|
||||
|
||||
def joins(fields = all.map { _1['field'] })
|
||||
result = ''
|
||||
if fields.include?('tags')
|
||||
result += <<~SQL.squish
|
||||
JOIN JSON_TABLE(CAST(v.tags_json AS JSON), '$[*]' COLUMNS (
|
||||
tag_id BIGINT PATH '$.tag_id' NULL ON EMPTY,
|
||||
external_tag_id BIGINT PATH '$.external_tag_id' NULL ON EMPTY)) tag_entry
|
||||
|
||||
SQL
|
||||
end
|
||||
if fields.include?('changed_field')
|
||||
fields = @definition.changes.map do |field|
|
||||
"SELECT #{ ApplicationRecord.connection.quote(field) } AS field_key"
|
||||
end.join(' UNION ALL ')
|
||||
condition = @definition.changes.map do |field|
|
||||
"(changed.field_key = '#{ field }' AND #{ @population.changed_sql(field) })"
|
||||
end.join(' OR ')
|
||||
unless HistoryAccess.admin?
|
||||
condition = "(#{ condition }) OR NOT (#{ HistoryAccess.sql(@definition.model, 'v') })"
|
||||
end
|
||||
result += " JOIN (#{ fields }) changed ON (#{ condition })"
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def parse(value)
|
||||
axes = QueryState.axes(value)
|
||||
raise ArgumentError unless axes.is_a?(Array)
|
||||
|
||||
axes.each do |axis|
|
||||
raise ArgumentError unless axis.is_a?(Hash) && (axis.keys - ['field', 'grain']).empty?
|
||||
raise ArgumentError unless @definition.axes(@population.history).include?(axis['field'])
|
||||
if @definition.fields[axis['field']] == :datetime
|
||||
raise ArgumentError unless GRAINS.include?(axis.fetch('grain', 'month'))
|
||||
|
||||
axis['grain'] ||= 'month'
|
||||
elsif axis.key?('grain')
|
||||
raise ArgumentError
|
||||
end
|
||||
end
|
||||
axes
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,59 @@
|
||||
# Bounded, authorised candidates at the same recorded point as the Counts query.
|
||||
class CountsCandidates
|
||||
KEYS = ['resource', 'field', 'query', 'category', 'page', 'locale', 'at',
|
||||
'execution_at', 'timezone'].freeze
|
||||
LIMIT = 20
|
||||
|
||||
def initialize(params)
|
||||
QueryState.validate!(params, KEYS)
|
||||
raise ArgumentError unless params.fetch('resource') == 'posts'
|
||||
raise ArgumentError unless ['tags', 'tags_json'].include?(params.fetch('field'))
|
||||
|
||||
@params = params
|
||||
@page = Integer(params.fetch('page', '1'), 10)
|
||||
raise ArgumentError unless @page.between?(1, 100_000)
|
||||
raise ArgumentError if params.fetch('query', '').bytesize > 1000
|
||||
category = params['category']
|
||||
raise ArgumentError if category.present? && !Tag.categories.key?(category)
|
||||
end
|
||||
|
||||
def call
|
||||
options = @params.slice('locale', 'at', 'execution_at', 'timezone')
|
||||
.merge('definition' => 'latest', 'resource' => 'tags')
|
||||
internal = CountsPopulation.new(options)
|
||||
external = CountsPopulation.new(options.merge('resource' => 'nico_tags',
|
||||
'execution_at' => internal.at.iso8601(6)))
|
||||
connection = TagVersion.connection
|
||||
sources = [[internal, 'internal', 'v.category', 'v.names'],
|
||||
[external, 'external', 'NULL', 'JSON_OBJECT()']]
|
||||
sources = sources.map do |population, kind, category, names|
|
||||
scope = population.scope.except(:select)
|
||||
if @params['query'].present?
|
||||
pattern = "%#{ ApplicationRecord.sanitize_sql_like(@params['query']) }%"
|
||||
scope = scope.where("(v.name LIKE ? OR CAST(v.tag_id AS CHAR) = ? OR EXISTS (" +
|
||||
"SELECT 1 FROM JSON_TABLE(COALESCE(JSON_EXTRACT(#{ names }, '$**.name'), " +
|
||||
"JSON_ARRAY()), '$[*]' COLUMNS (name TEXT PATH '$')) candidate_name " +
|
||||
'WHERE candidate_name.name LIKE ?))', pattern, @params['query'], pattern)
|
||||
end
|
||||
scope.select(Arel.sql(
|
||||
"CONCAT('#{ kind }:', v.tag_id) AS value, v.tag_id AS id, " +
|
||||
"v.name AS name, #{ names } AS names, #{ category } AS category, '#{ kind }' AS kind"))
|
||||
.to_sql
|
||||
end
|
||||
clauses = []
|
||||
binds = []
|
||||
if @params['category'].present?
|
||||
clauses << 'category = ?'
|
||||
binds << @params['category']
|
||||
end
|
||||
where = clauses.empty? ? '' : "WHERE #{ clauses.join(' AND ') }"
|
||||
sql = <<~SQL
|
||||
SELECT /*+ MAX_EXECUTION_TIME(10000) */ * FROM (#{ sources.join(' UNION ALL ') }) choices
|
||||
#{ where } ORDER BY kind, id LIMIT #{ LIMIT + 1 } OFFSET #{ (@page - 1) * LIMIT }
|
||||
SQL
|
||||
rows = connection.select_all(ApplicationRecord.sanitize_sql_array([sql, *binds])).to_a
|
||||
rows.each { _1['names'] = JSON.parse(_1['names']) if _1['names'].is_a?(String) }
|
||||
{ rows: rows.first(LIMIT), page: @page, more: rows.length > LIMIT,
|
||||
execution_at: internal.at.iso8601(6) }
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,473 @@
|
||||
# Typed predicates over the same recorded fields used by Counts output.
|
||||
# Multi-valued comparisons use EXISTS; inequality means no matching element.
|
||||
class CountsCondition
|
||||
CHILD_TYPES = { 'id' => :integer, 'versionNo' => :integer, 'tagId' => :integer,
|
||||
'tagGroupId' => :integer, 'autoGenerated' => :boolean,
|
||||
'category' => :enum, 'kind' => :enum, 'sections' => :interval }.freeze
|
||||
|
||||
def self.fields(definition)
|
||||
result = definition.metadata.map { _1.merge(labels: [definition.field_label(_1[:key])]) }
|
||||
visit = lambda do |fields, prefix, labels|
|
||||
fields.each do |field|
|
||||
key = [prefix, field[:key]].compact.join('.')
|
||||
path_labels = labels + [field[:label]]
|
||||
type = if prefix
|
||||
CHILD_TYPES.fetch(field[:key], :string)
|
||||
elsif key == 'latest'
|
||||
:boolean
|
||||
elsif key == 'parents'
|
||||
:integer
|
||||
elsif key == 'deprecated'
|
||||
:datetime
|
||||
elsif key == 'thumbnail'
|
||||
:string
|
||||
else
|
||||
definition.fields.fetch(key, :string)
|
||||
end
|
||||
type = :string if field[:children] || type == :json
|
||||
type = :integer if ['parents', 'members'].include?(key)
|
||||
physical = result.find { _1[:key] == key }
|
||||
nullable = prefix.present? || (physical ? physical[:nullable] : key != 'latest')
|
||||
nullable = false if field[:children]
|
||||
entry = { key:, type:, labels: path_labels, nullable:,
|
||||
multiple: prefix.present? || field[:children].present? || field[:multiple_values],
|
||||
collection: field[:children].present? || type == :interval,
|
||||
operators: CountsResourceDefinition::OPERATORS.fetch(type) }
|
||||
entry[:label_values] = field[:label_values] if field[:label_values]
|
||||
entry[:name_locale] = field[:name_locale] if field[:name_locale]
|
||||
entry[:legacy] = true if field[:legacy]
|
||||
result.reject! { _1[:key] == key }
|
||||
result << entry
|
||||
visit.call(field[:children] || [], key, path_labels)
|
||||
end
|
||||
end
|
||||
visit.call(definition.condition_fields, nil, [])
|
||||
# The raw JSON alias has the same membership semantics as the public tag field.
|
||||
if definition.resource == 'posts'
|
||||
result.find { _1[:key] == 'tags_json' }.merge!(type: :string, multiple: true, collection: true,
|
||||
operators: CountsResourceDefinition::OPERATORS.fetch(:string))
|
||||
end
|
||||
result += [{ key: 'user.id', type: :integer, nullable: true,
|
||||
labels: ['history:actor', 'counts:columns.resourceId'],
|
||||
operators: CountsResourceDefinition::OPERATORS.fetch(:integer) },
|
||||
{ key: 'user.name', type: :string, nullable: true,
|
||||
labels: ['history:actor', 'history:name'],
|
||||
operators: CountsResourceDefinition::OPERATORS.fetch(:string) }]
|
||||
result.each do |field|
|
||||
next if field[:type] == :interval
|
||||
|
||||
if field[:multiple]
|
||||
field[:operators] = field[:operators] - ['in', 'not_in'] + ['any', 'all', 'none']
|
||||
end
|
||||
field[:input_types] = ['single', 'multiple']
|
||||
if ['category', 'tag_category'].include?(field[:key].split('.').last)
|
||||
field[:candidate_source] = 'categories'
|
||||
field[:choices] = Tag.categories.keys.map { |value|
|
||||
{ value:, label: "common:categories.#{ value }" }
|
||||
}
|
||||
elsif definition.resource == 'posts' && ['tags', 'tags_json'].include?(field[:key])
|
||||
field[:candidate_source] = 'tags'
|
||||
field[:set_type] = 'tag_reference'
|
||||
end
|
||||
field[:input_types] << 'list' if field[:candidate_source]
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
def initialize(population)
|
||||
@population = population
|
||||
@definition = population.definition
|
||||
@connection = @definition.model.connection
|
||||
end
|
||||
|
||||
def apply(scope, entry)
|
||||
raise ArgumentError unless entry.is_a?(Hash)
|
||||
if entry['operator'] == 'axis_eq'
|
||||
raise ArgumentError unless (entry.keys - ['field', 'operator', 'grain', 'value']).empty?
|
||||
value = entry['value']
|
||||
unless value.nil? || value.is_a?(String) || value.is_a?(Numeric) || [true, false].include?(value)
|
||||
raise ArgumentError
|
||||
end
|
||||
raise ArgumentError if value.is_a?(String) && value.bytesize > 8192
|
||||
|
||||
return @population.axis_condition(scope, entry)
|
||||
end
|
||||
allowed = ['field', 'operator', 'value', 'end', 'blank', 'input', 'values']
|
||||
raise ArgumentError if (entry.keys - allowed).any?
|
||||
|
||||
field = @definition.filter_metadata.find { _1[:key] == entry.fetch('field') }
|
||||
raise ArgumentError unless field && field[:operators].include?(entry.fetch('operator'))
|
||||
raise ArgumentError unless [true, false, nil].include?(entry['blank'])
|
||||
input = entry.fetch('input', 'single')
|
||||
raise ArgumentError unless ['single', 'multiple', 'list'].include?(input)
|
||||
if input == 'list' && !field[:candidate_source]
|
||||
raise ArgumentError
|
||||
end
|
||||
values = [entry['value'], entry['end']]
|
||||
raise ArgumentError unless values.all? { _1.nil? || (_1.is_a?(String) && _1.bytesize <= 8192) }
|
||||
if entry['operator'] != 'range' && !values[1].nil? && values[1] != ''
|
||||
raise ArgumentError
|
||||
end
|
||||
|
||||
source, expression = source_for(field[:key])
|
||||
type = field[:type]
|
||||
blank = blank_sql(expression, type)
|
||||
if entry['blank']
|
||||
sql = if source && field[:collection]
|
||||
"NOT EXISTS (SELECT 1 FROM #{ source })"
|
||||
elsif source
|
||||
"NOT EXISTS (SELECT 1 FROM #{ source } WHERE NOT (#{ blank }))"
|
||||
else blank
|
||||
end
|
||||
return scope.where(Arel.sql(sql))
|
||||
end
|
||||
if ['in', 'not_in', 'any', 'all', 'none'].include?(entry['operator'])
|
||||
return set_condition(scope, entry, field)
|
||||
end
|
||||
raise ArgumentError if entry.key?('values') || input != 'single'
|
||||
return scope if values.all? { _1.nil? || _1 == '' }
|
||||
|
||||
parsed = values.map { parse(_1, type) }
|
||||
validate_enum!(field[:key], parsed.first) if type == :enum
|
||||
operator = entry['operator']
|
||||
if type == :interval
|
||||
return interval(scope, source, expression, parsed)
|
||||
end
|
||||
clauses = []
|
||||
binds = []
|
||||
if operator == 'range'
|
||||
['>=', '<='].each_with_index do |comparison, index|
|
||||
next if parsed[index].nil?
|
||||
|
||||
clauses << "#{ expression } #{ comparison } ?"
|
||||
binds << parsed[index]
|
||||
end
|
||||
raise ArgumentError if parsed.all? && parsed.first > parsed.last
|
||||
elsif !parsed.first.nil?
|
||||
comparison = { 'eq' => '=', 'ne' => '=', 'gt' => '>', 'lt' => '<',
|
||||
'ge' => '>=', 'le' => '<=', 'like' => 'LIKE' }.fetch(operator)
|
||||
clauses << "#{ expression } #{ comparison } ?"
|
||||
binds << parsed.first
|
||||
end
|
||||
return scope if clauses.empty?
|
||||
|
||||
predicate = clauses.join(' AND ')
|
||||
if source
|
||||
predicate = "EXISTS (SELECT 1 FROM #{ source } WHERE #{ predicate })"
|
||||
predicate = "NOT (#{ predicate })" if operator == 'ne'
|
||||
elsif operator == 'ne'
|
||||
predicate = "(NOT (#{ predicate }) OR #{ blank })"
|
||||
end
|
||||
scope.where(predicate, *binds)
|
||||
end
|
||||
|
||||
def tag_name_match(value)
|
||||
source = post_tag_source
|
||||
bound = @population.history ? 'v.created_at' : @connection.quote(point)
|
||||
source += ' LEFT JOIN nico_tag_versions external_snapshot ON external_snapshot.tag_id = ' +
|
||||
scalar('tag_snapshot.item', 'external_tag_id', :integer) +
|
||||
" AND external_snapshot.created_at <= #{ bound } AND " +
|
||||
@population.access_sql(NicoTagVersion, 'external_snapshot') +
|
||||
' AND NOT EXISTS (SELECT 1 FROM nico_tag_versions newer_external ' +
|
||||
'WHERE newer_external.tag_id = external_snapshot.tag_id ' +
|
||||
"AND newer_external.created_at <= #{ bound } " +
|
||||
'AND newer_external.version_no > external_snapshot.version_no)'
|
||||
document = "JSON_MERGE_PRESERVE(JSON_ARRAY(#{ scalar('tag_snapshot.item', 'name') }, " +
|
||||
"referenced_tag.name, CONCAT('nico:', external_snapshot.name)), " +
|
||||
"COALESCE(JSON_EXTRACT(referenced_tag.names, '$**.name'), JSON_ARRAY()))"
|
||||
source += ' JOIN ' + json_table(document, '$[*]', 'search_name')
|
||||
@definition.model.sanitize_sql_array([
|
||||
"EXISTS (SELECT 1 FROM #{ source } WHERE JSON_UNQUOTE(search_name.item) = ?)", value])
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def set_condition(scope, entry, field)
|
||||
return scope unless entry.key?('values')
|
||||
|
||||
values = entry['values']
|
||||
raise ArgumentError unless values.is_a?(Array) && values.length <= 100
|
||||
raise ArgumentError if [entry['value'], entry['end']].any? { !_1.nil? && _1 != '' }
|
||||
|
||||
parsed = values.map do |value|
|
||||
unless (value.is_a?(String) || value.is_a?(Numeric) || [true, false].include?(value))
|
||||
raise ArgumentError
|
||||
end
|
||||
raise ArgumentError if value.to_s.bytesize > 8192
|
||||
|
||||
if field[:set_type] == 'tag_reference'
|
||||
raise ArgumentError unless value.is_a?(String) &&
|
||||
value.match?(/\A(?:internal|external):[1-9][0-9]{0,17}\z/)
|
||||
value
|
||||
else
|
||||
raise ArgumentError if value == '' && ![:string, :enum].include?(field[:type])
|
||||
result = value == '' ? '' : parse(value.to_s, field[:type])
|
||||
validate_enum!(field[:key], result) if field[:type] == :enum
|
||||
result
|
||||
end
|
||||
end
|
||||
raise ArgumentError unless parsed.uniq == parsed
|
||||
|
||||
operator = entry['operator']
|
||||
if parsed.empty?
|
||||
return ['in', 'any'].include?(operator) ? scope.none : scope
|
||||
end
|
||||
source, expression = source_for(field[:key])
|
||||
if field[:set_type] == 'tag_reference'
|
||||
source = post_tag_source
|
||||
id = "COALESCE(#{ scalar('tag_snapshot.item', 'tag_id', :integer) }, " +
|
||||
"#{ scalar('tag_snapshot.item', 'external_tag_id', :integer) })"
|
||||
expression = "CONCAT(CASE WHEN #{ scalar('tag_snapshot.item', 'tag_id') } IS NULL " +
|
||||
"THEN 'external:' ELSE 'internal:' END, #{ id })"
|
||||
end
|
||||
predicates = parsed.map do |_value|
|
||||
clause = "#{ expression } = ?"
|
||||
clause = "EXISTS (SELECT 1 FROM #{ source } WHERE #{ clause })" if source
|
||||
clause
|
||||
end
|
||||
conjunction = operator == 'all' ? ' AND ' : ' OR '
|
||||
predicate = "(#{ predicates.join(conjunction) })"
|
||||
if ['not_in', 'none'].include?(operator)
|
||||
predicate = source ? "NOT #{ predicate }" :
|
||||
"(NOT #{ predicate } OR #{ blank_sql(expression, field[:type]) })"
|
||||
end
|
||||
scope.where(predicate, *parsed)
|
||||
end
|
||||
|
||||
def parse(value, type)
|
||||
return nil if value.nil? || value == ''
|
||||
|
||||
case type
|
||||
when :integer, :interval
|
||||
raise ArgumentError unless value.match?(/\A-?[0-9]{1,19}\z/)
|
||||
|
||||
number = Integer(value, 10)
|
||||
raise ArgumentError unless number.between?(-9_223_372_036_854_775_808, 9_223_372_036_854_775_807)
|
||||
|
||||
number
|
||||
when :boolean
|
||||
raise ArgumentError unless ['true', 'false'].include?(value)
|
||||
|
||||
value == 'true' ? 1 : 0
|
||||
when :datetime then @population.timestamp(value)
|
||||
else value
|
||||
end
|
||||
end
|
||||
|
||||
def validate_enum!(key, value)
|
||||
allowed = case key.split('.').last
|
||||
when 'event_type' then VersionRecord::DEFAULT_EVENT_TYPE_MAP.values
|
||||
when 'category', 'tag_category' then TagVersion.categories.keys
|
||||
when 'kind' then ['internal', 'external']
|
||||
when 'source_kind' then Material::SOURCE_KINDS
|
||||
end
|
||||
raise ArgumentError if value && allowed && !allowed.include?(value)
|
||||
end
|
||||
|
||||
def blank_sql(expression, type)
|
||||
if [:string, :enum].include?(type)
|
||||
"(#{ expression } IS NULL OR #{ expression } = '')"
|
||||
else
|
||||
"#{ expression } IS NULL"
|
||||
end
|
||||
end
|
||||
|
||||
def scalar(document, key, type = :string)
|
||||
path = @connection.quote("$.#{ key }")
|
||||
value = "JSON_EXTRACT(#{ document }, #{ path })"
|
||||
text = "JSON_UNQUOTE(NULLIF(#{ value }, CAST('null' AS JSON)))"
|
||||
return "CAST(#{ text } AS SIGNED)" if type == :integer
|
||||
return "CASE #{ text } WHEN 'true' THEN 1 WHEN 'false' THEN 0 END" if type == :boolean
|
||||
|
||||
text
|
||||
end
|
||||
|
||||
def json_table(document, path, name)
|
||||
"JSON_TABLE(COALESCE(#{ document }, JSON_ARRAY()), #{ @connection.quote(path) } " +
|
||||
"COLUMNS (item JSON PATH '$')) #{ name }"
|
||||
end
|
||||
|
||||
def names_source(document, alias_name)
|
||||
source = "JSON_TABLE(COALESCE(JSON_KEYS(#{ document }), JSON_ARRAY()), '$[*]' " +
|
||||
"COLUMNS (language_code TEXT PATH '$')) #{ alias_name }"
|
||||
path = "CONCAT('$.', JSON_QUOTE(#{ alias_name }.language_code))"
|
||||
[source, "JSON_EXTRACT(#{ document }, #{ path })", "#{ alias_name }.language_code"]
|
||||
end
|
||||
|
||||
def point
|
||||
@point ||= @population.history ? @population.at :
|
||||
@population.timestamp(@population.params['at']) || @population.at
|
||||
end
|
||||
|
||||
def name_source(row, name)
|
||||
document = "JSON_MERGE_PRESERVE(JSON_ARRAY(#{ row }.name), " +
|
||||
"COALESCE(JSON_EXTRACT(#{ row }.names, '$.*.name'), JSON_ARRAY()))"
|
||||
[json_table(document, '$[*]', name), "JSON_UNQUOTE(#{ name }.item)"]
|
||||
end
|
||||
|
||||
def parent_source
|
||||
column = @definition.resource == 'posts' ? 'parent_post_ids' : 'parent_tag_ids'
|
||||
ids = "TRIM(REGEXP_REPLACE(COALESCE(v.#{ column }, ''), '[[:space:]]+', ' '))"
|
||||
document = "CONCAT('[', REPLACE(#{ ids }, ' ', ','), ']')"
|
||||
source = json_table(document, '$[*]', 'parent_entry')
|
||||
id = 'CAST(parent_entry.item AS UNSIGNED)'
|
||||
return [source, id] if @definition.resource == 'posts'
|
||||
|
||||
bound = @population.history ? 'v.created_at' : @connection.quote(point)
|
||||
source += ' LEFT JOIN tag_versions parent_version ON parent_version.tag_id = ' + id +
|
||||
" AND parent_version.created_at <= #{ bound } AND " +
|
||||
@population.access_sql(TagVersion, 'parent_version') +
|
||||
' AND NOT EXISTS (SELECT 1 FROM tag_versions newer_parent ' +
|
||||
'WHERE newer_parent.tag_id = parent_version.tag_id ' +
|
||||
"AND newer_parent.created_at <= #{ bound } " +
|
||||
'AND newer_parent.version_no > parent_version.version_no)'
|
||||
[source, id]
|
||||
end
|
||||
|
||||
def post_tag_source
|
||||
source = json_table('v.tags_json', '$[*]', 'tag_snapshot')
|
||||
source += ' LEFT JOIN tag_versions referenced_tag ON referenced_tag.tag_id = ' +
|
||||
scalar('tag_snapshot.item', 'tag_id', :integer) +
|
||||
' AND referenced_tag.version_no = ' + scalar('tag_snapshot.item', 'version_no', :integer) +
|
||||
" AND referenced_tag.created_at <= #{ @population.history ? 'v.created_at' :
|
||||
@connection.quote(point) } AND " +
|
||||
@population.access_sql(TagVersion, 'referenced_tag')
|
||||
source
|
||||
end
|
||||
|
||||
def language_field(source, document, parts, inherited_language = nil)
|
||||
return [source, scalar(document, 'name')] if parts.empty?
|
||||
return [source, inherited_language] if parts == ['languageCode'] && inherited_language
|
||||
|
||||
if parts.first == 'aliases'
|
||||
aliases = "JSON_EXTRACT(#{ document }, '$.aliases')"
|
||||
source += ' JOIN ' + json_table(aliases, '$[*]', 'alias_entry')
|
||||
child = parts[1] || 'name'
|
||||
return [source, inherited_language] if child == 'languageCode' && inherited_language
|
||||
columns = { 'scriptCode' => 'script_code', 'name' => 'name' }
|
||||
key = columns.fetch(child) { raise ArgumentError }
|
||||
return [source, scalar('alias_entry.item', key)]
|
||||
end
|
||||
key = { 'name' => 'name', 'scriptCode' => 'script_code',
|
||||
'autoGenerated' => 'auto_generated' }.fetch(parts.first) { raise ArgumentError }
|
||||
[source, scalar(document, key, CHILD_TYPES.fetch(parts.first, :string))]
|
||||
end
|
||||
|
||||
def source_for(key)
|
||||
root, *parts = key.split('.')
|
||||
if @definition.resource == 'tags' && @definition.name_column(root)
|
||||
return @definition.alias_source(root) if root.start_with?('aliases_')
|
||||
return [nil, @definition.name_expression(root)]
|
||||
end
|
||||
return [nil, "(#{ @population.latest_sql })"] if root == 'latest'
|
||||
if root == 'user' && parts.any?
|
||||
expression = parts.first == 'id' ? 'v.created_by_user_id' :
|
||||
'(SELECT actor.name FROM users actor WHERE actor.id = v.created_by_user_id)'
|
||||
return [nil, expression]
|
||||
end
|
||||
return [nil, 'v.thumbnail_base'] if root == 'thumbnail'
|
||||
return [nil, 'v.deprecated_at'] if root == 'deprecated'
|
||||
if root == 'parents'
|
||||
source, id = parent_source
|
||||
if @definition.resource == 'tags' && @definition.name_column(parts.first)
|
||||
if parts.first.start_with?('aliases_')
|
||||
aliases, value = @definition.alias_source(parts.first, 'parent_version')
|
||||
return ["#{ source } JOIN #{ aliases }", value]
|
||||
end
|
||||
return [source, @definition.name_expression(parts.first, 'parent_version')]
|
||||
end
|
||||
if parts.first == 'names'
|
||||
names, document, language = names_source('parent_version.names', 'parent_language')
|
||||
return language_field("#{ source } JOIN #{ names }", document, parts.drop(1), language)
|
||||
end
|
||||
return [source, id] unless parts.first == 'name'
|
||||
|
||||
names, value = name_source('parent_version', 'parent_name')
|
||||
return ["#{ source } JOIN #{ names }", value]
|
||||
end
|
||||
if root == 'tags' || root == 'tags_json'
|
||||
return post_field(parts)
|
||||
end
|
||||
if root == 'names' || root == 'aliases'
|
||||
source, document, language = names_source('v.names', 'name_language')
|
||||
parts = ['aliases', *parts] if root == 'aliases'
|
||||
return language_field(source, document, parts, language)
|
||||
end
|
||||
if root == 'members' || root == 'placements'
|
||||
path = '$[*]' + parts.take_while { _1 == 'children' }.map { '.children[*]' }.join
|
||||
source = json_table("v.#{ root }", path, 'structure_entry')
|
||||
if root == 'members' && parts.empty?
|
||||
value = "COALESCE(#{ scalar('structure_entry.item', 'tag_id', :integer) }, " +
|
||||
"#{ scalar('structure_entry.item', 'tag_group_id', :integer) })"
|
||||
return [source, value]
|
||||
end
|
||||
leaf = parts.last || 'sectionName'
|
||||
leaf = 'sectionName' if leaf == 'children'
|
||||
column = leaf.gsub(/[A-Z]/) { "_#{ _1.downcase }" }
|
||||
return [source, scalar('structure_entry.item', column, CHILD_TYPES.fetch(leaf, :string))]
|
||||
end
|
||||
if root == 'export_paths_json'
|
||||
source, value, profile = names_source('v.export_paths_json', 'export_profile')
|
||||
return [source, parts.first == 'profile' ? profile : "JSON_UNQUOTE(#{ value })"]
|
||||
end
|
||||
return name_source('v', 'recorded_name') if root == 'name' && @definition.resource == 'tags'
|
||||
|
||||
[nil, @definition.expression(root)]
|
||||
end
|
||||
|
||||
def post_field(parts)
|
||||
source = post_tag_source
|
||||
document = 'tag_snapshot.item'
|
||||
if @definition.name_column(parts.first)
|
||||
if parts.first.start_with?('aliases_')
|
||||
aliases, value = @definition.alias_source(parts.first, 'referenced_tag')
|
||||
return ["#{ source } JOIN #{ aliases }", value]
|
||||
end
|
||||
return [source, @definition.name_expression(parts.first, 'referenced_tag')]
|
||||
end
|
||||
case parts.first
|
||||
when 'names', 'aliases'
|
||||
names, entry, language = names_source('referenced_tag.names', 'tag_language')
|
||||
fields = parts.first == 'aliases' ? ['aliases', *parts.drop(1)] : parts.drop(1)
|
||||
language_field("#{ source } JOIN #{ names }", entry, fields, language)
|
||||
when 'sections'
|
||||
sections = "JSON_EXTRACT(#{ document }, '$.sections')"
|
||||
source += ' JOIN ' + json_table(sections, '$[*]', 'section_entry')
|
||||
[source, 'section_entry.item']
|
||||
when 'id'
|
||||
[source, "COALESCE(#{ scalar(document, 'tag_id', :integer) }, " +
|
||||
"#{ scalar(document, 'external_tag_id', :integer) })"]
|
||||
when 'versionNo' then [source, scalar(document, 'version_no', :integer)]
|
||||
when 'kind'
|
||||
[source, "CASE WHEN #{ scalar(document, 'tag_id') } IS NULL " +
|
||||
"THEN 'external' ELSE 'internal' END"]
|
||||
when 'category'
|
||||
[source, "COALESCE(#{ scalar(document, 'category') }, referenced_tag.category)"]
|
||||
else
|
||||
names = "JSON_MERGE_PRESERVE(JSON_ARRAY(#{ scalar(document, 'name') }, " +
|
||||
'referenced_tag.name), ' +
|
||||
"COALESCE(JSON_EXTRACT(referenced_tag.names, '$.*.name'), JSON_ARRAY()))"
|
||||
["#{ source } JOIN #{ json_table(names, '$[*]', 'recorded_name') }",
|
||||
'JSON_UNQUOTE(recorded_name.item)']
|
||||
end
|
||||
end
|
||||
|
||||
def interval(scope, source, document, bounds)
|
||||
raise ArgumentError if bounds.compact.any?(&:negative?)
|
||||
raise ArgumentError if bounds.all? && bounds.first > bounds.last
|
||||
|
||||
first = scalar(document, 'begin_ms', :integer)
|
||||
last = scalar(document, 'end_ms', :integer)
|
||||
clauses = []
|
||||
values = []
|
||||
if bounds.first
|
||||
clauses << "(#{ last } IS NULL OR #{ last } > ?)"
|
||||
values << bounds.first
|
||||
end
|
||||
if bounds.last
|
||||
clauses << "#{ first } < ?"
|
||||
values << bounds.last
|
||||
end
|
||||
scope.where("EXISTS (SELECT 1 FROM #{ source } WHERE #{ clauses.join(' AND ') })", *values)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,94 @@
|
||||
# Only confirmed combinations belong here; all others retain the existing behaviour.
|
||||
class CountsDefaults
|
||||
POST_COLUMNS = ['resource_id', 'version_no', 'latest', 'title', 'url',
|
||||
'thumbnail_base', 'tags', 'parents', 'original_created_from',
|
||||
'original_created_before', 'event_type', 'user', 'created_at'].freeze
|
||||
|
||||
def self.entries(resource, locale: 'ja')
|
||||
result = [{ definition: 'history', mode: 'summary', params: {
|
||||
'rows' => 'created_at:month',
|
||||
'cols' => 'event_type',
|
||||
'sort' => 'a0:desc', 'conditions' => '' } }]
|
||||
result << { definition: 'latest', mode: 'summary', params: {
|
||||
'rows' => '', 'cols' => '', 'conditions' => '', 'tags' => '' } }
|
||||
details = detail_params(resource)
|
||||
if details
|
||||
details = details.merge('sort' => 'resource_id:asc version_no:asc')
|
||||
language = Locale.find_by(code: locale)&.language_code || locale
|
||||
name = CountsResourceDefinition.name_key(language)
|
||||
aliases = CountsResourceDefinition.aliases_key(language)
|
||||
details = details.transform_values do |value|
|
||||
value.split(' ').flat_map do |token|
|
||||
if resource == 'tags' && ['name', 'parents.name'].include?(token)
|
||||
token == 'name' ? name : ["parents.#{ name }", "parents.#{ aliases }"]
|
||||
elsif resource == 'tags' && token == 'aliases'
|
||||
aliases
|
||||
elsif resource == 'tags' && token.start_with?('aliases.')
|
||||
[]
|
||||
elsif resource == 'posts' && token == 'tags.name'
|
||||
"tags.#{ name }"
|
||||
else token
|
||||
end
|
||||
end.join(' ')
|
||||
end
|
||||
if ['posts', 'tags'].include?(resource)
|
||||
fields = CountsResourceDefinition.new(resource).output_fields
|
||||
selected = details.fetch('columns').split(' ')
|
||||
details['columns'] = fields.map { _1[:key] }.select { selected.include?(_1) }.join(' ')
|
||||
order = lambda do |items, prefix|
|
||||
items.flat_map do |field|
|
||||
path = [prefix, field[:key]].compact.join('.')
|
||||
[path, *order.call(field.fetch(:children, []), path)]
|
||||
end
|
||||
end
|
||||
details['order'] = order.call(fields, nil).join(' ')
|
||||
end
|
||||
['latest', 'history'].each do |definition|
|
||||
result << { definition:, mode: 'details', params: details }
|
||||
end
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
def self.detail_params(resource)
|
||||
if resource == 'posts'
|
||||
return { 'columns' => POST_COLUMNS.join(' '), 'order' => POST_COLUMNS.join(' '),
|
||||
'children' => 'tags.name parents.id', 'nested_json' => '0',
|
||||
'conditions' => '', 'tags' => '' }
|
||||
end
|
||||
unless resource == 'tags'
|
||||
return { 'columns' => CountsExportProjection.default_columns(resource).join(' '),
|
||||
'children' => CountsExportProjection.default_children(resource).join(' ') }
|
||||
end
|
||||
|
||||
columns = ['resource_id', 'version_no', 'latest', 'name', 'category',
|
||||
'aliases', 'parents', 'deprecated', 'event_type', 'user', 'created_at']
|
||||
{ 'columns' => columns.join(' '), 'order' => columns.join(' '),
|
||||
'children' => 'aliases.name parents.name', 'nested_json' => '0' }
|
||||
end
|
||||
|
||||
def self.for(params)
|
||||
defaults = entries(params.fetch('resource', 'posts'), locale: params.fetch('locale', 'ja'))
|
||||
settings = defaults.find do |entry|
|
||||
entry[:definition] == params.fetch('definition', 'latest') &&
|
||||
entry[:mode] == params.fetch('mode', 'summary')
|
||||
end&.fetch(:params) || {}
|
||||
if settings.key?('sort') && params.key?('rows') && params.fetch('mode', 'summary') == 'summary'
|
||||
actual = QueryState.axes(params['rows'])
|
||||
expected = QueryState.axes(settings.fetch('rows'))
|
||||
compatible = actual.is_a?(Array) && actual.first.is_a?(Hash) &&
|
||||
actual.first['field'] == expected.first['field'] &&
|
||||
actual.first.fetch('grain', 'month') == expected.first['grain']
|
||||
unless compatible
|
||||
settings = settings.except('sort')
|
||||
end
|
||||
end
|
||||
settings
|
||||
end
|
||||
|
||||
def self.metadata(resource, locale: 'ja')
|
||||
entries(resource, locale:).map do |entry|
|
||||
entry.merge(params: entry[:params].map { |key, value| { key:, value: } })
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,107 @@
|
||||
require 'tempfile'
|
||||
require 'fileutils'
|
||||
|
||||
# Finish generation before reporting readiness; downloads never hold the full file in RAM.
|
||||
class CountsExportDownload
|
||||
EXPIRY = 10.minutes
|
||||
MAX_CONCURRENT_EXPORTS = 2
|
||||
MAX_FILE_BYTES = 128.megabytes
|
||||
MAX_STORED_BYTES = 512.megabytes
|
||||
MAX_EXPORT_SECONDS = 120
|
||||
class Busy < StandardError; end
|
||||
class TooLarge < StandardError; end
|
||||
class TimedOut < StandardError; end
|
||||
class StorageFull < StandardError; end
|
||||
|
||||
def self.directory
|
||||
Rails.root.join(ENV.fetch('COUNTS_EXPORT_DIRECTORY', 'tmp/counts_exports'))
|
||||
end
|
||||
def self.verifier = Rails.application.message_verifier('counts_export_download')
|
||||
|
||||
def self.prepare(exporter, owner, progress: nil, phase: nil, check: nil, identifier: nil)
|
||||
FileUtils.mkdir_p(directory, mode: 0o700)
|
||||
authority = HistoryAccess.authority
|
||||
slot = acquire_slot
|
||||
|
||||
begin
|
||||
generate(exporter, owner, progress:, phase:, check:, identifier:, authority:)
|
||||
ensure
|
||||
slot.close
|
||||
end
|
||||
end
|
||||
|
||||
def self.acquire_slot
|
||||
MAX_CONCURRENT_EXPORTS.times do |index|
|
||||
path = directory.join(".slot-#{ index }.lock")
|
||||
handle = File.open(path, File::RDWR | File::CREAT, 0o600)
|
||||
return handle if handle.flock(File::LOCK_EX | File::LOCK_NB)
|
||||
|
||||
handle.close
|
||||
end
|
||||
raise Busy
|
||||
end
|
||||
|
||||
def self.generate(exporter, owner, progress: nil, phase: nil, check: nil,
|
||||
identifier: nil, authority: HistoryAccess.authority)
|
||||
FileUtils.mkdir_p(directory, mode: 0o700)
|
||||
Dir.glob(directory.join('counts-*.export')).each do |path|
|
||||
File.unlink(path) if File.mtime(path) < Time.current - EXPIRY
|
||||
rescue Errno::ENOENT
|
||||
# Another request may have removed an expired download.
|
||||
end
|
||||
stored = Dir.glob(directory.join('counts-*')).sum do |path|
|
||||
File.size(path)
|
||||
rescue Errno::ENOENT
|
||||
0
|
||||
end
|
||||
reserved = MAX_CONCURRENT_EXPORTS * MAX_FILE_BYTES
|
||||
raise StorageFull if stored + reserved > MAX_STORED_BYTES
|
||||
|
||||
prefix = identifier ? "counts-#{ identifier }-" : 'counts-'
|
||||
file = Tempfile.create([prefix, '.part'], directory)
|
||||
complete = false
|
||||
bytes = 0
|
||||
deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + MAX_EXPORT_SECONDS
|
||||
path = file.path
|
||||
begin
|
||||
file.binmode
|
||||
exporter.body(progress:, phase:).each do |chunk|
|
||||
check&.call
|
||||
raise TimedOut if Process.clock_gettime(Process::CLOCK_MONOTONIC) > deadline
|
||||
|
||||
bytes += chunk.bytesize
|
||||
raise TooLarge if bytes > MAX_FILE_BYTES
|
||||
|
||||
file.write(chunk)
|
||||
end
|
||||
file.flush
|
||||
file.fsync
|
||||
file.close
|
||||
finished_path = path.sub(/\.part\z/, '.export')
|
||||
File.rename(path, finished_path)
|
||||
path = finished_path
|
||||
token = verifier.generate({ 'file' => File.basename(path),
|
||||
'resource' => exporter.resource, 'format' => exporter.format,
|
||||
'owner' => owner, **authority }, expires_in: EXPIRY, purpose: 'counts_export')
|
||||
complete = true
|
||||
token
|
||||
ensure
|
||||
file.close unless file.closed?
|
||||
File.unlink(path) if !complete && File.exist?(path)
|
||||
end
|
||||
end
|
||||
|
||||
def self.find(token, owner)
|
||||
metadata = verifier.verified(token, purpose: 'counts_export')
|
||||
return unless metadata.is_a?(Hash) && metadata['owner'] == owner
|
||||
return unless HistoryAccess.authorised_artifact?(metadata)
|
||||
return unless metadata['file']&.match?(/\Acounts-[a-zA-Z0-9_-]+\.export\z/)
|
||||
return unless CountsResourceDefinition::DEFINITIONS.key?(metadata['resource'])
|
||||
return unless ['json', 'csv'].include?(metadata['format'])
|
||||
|
||||
path = directory.join(metadata['file'])
|
||||
return unless path.file?
|
||||
|
||||
metadata.merge('path' => path.to_s)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,412 @@
|
||||
require 'json'
|
||||
require 'fileutils'
|
||||
|
||||
# A durable, bounded export spool. Web and supervised workers share the same POSIX volume.
|
||||
class CountsExportJobs
|
||||
class Cancelled < StandardError; end
|
||||
class WorkerUnavailable < StandardError; end
|
||||
ACTIVE = ['queued', 'running', 'cancelling'].freeze
|
||||
HEARTBEAT_SECONDS = 15
|
||||
QUEUE_SECONDS = CountsExportDownload::MAX_EXPORT_SECONDS + 30
|
||||
|
||||
def self.worker_id
|
||||
@worker_id ||= ".worker-#{ Process.pid }-#{ SecureRandom.hex(8) }"
|
||||
end
|
||||
|
||||
def self.pulse
|
||||
FileUtils.mkdir_p(directory, mode: 0o700)
|
||||
write({ 'id' => worker_id, 'heartbeat_at' => Time.current.to_f })
|
||||
end
|
||||
|
||||
def self.worker_available?
|
||||
Dir.glob(directory.join('.worker-*.json')).any? do |path|
|
||||
JSON.parse(File.read(path)).fetch('heartbeat_at') > Time.current.to_f - HEARTBEAT_SECONDS
|
||||
rescue Errno::ENOENT
|
||||
false
|
||||
end
|
||||
end
|
||||
|
||||
def self.expire_stalled(state)
|
||||
now = Time.current.to_f
|
||||
queued = state['status'] == 'queued'
|
||||
stale = if queued
|
||||
now - state.fetch('created_at', now) > QUEUE_SECONDS ||
|
||||
(!worker_available? && now - state.fetch('created_at', now) > HEARTBEAT_SECONDS)
|
||||
else
|
||||
['running', 'cancelling'].include?(state['status']) &&
|
||||
now - state.fetch('heartbeat_at', now) > HEARTBEAT_SECONDS
|
||||
end
|
||||
return unless stale
|
||||
|
||||
state.merge!('status' => 'failed', 'error' => queued ? 'worker_unavailable' : 'worker_stalled',
|
||||
'updated_at' => now)
|
||||
remove_download(state)
|
||||
state.delete('token')
|
||||
write(state)
|
||||
end
|
||||
|
||||
def self.directory = CountsExportDownload.directory.join('jobs')
|
||||
def self.verifier = Rails.application.message_verifier('counts_export_job')
|
||||
|
||||
def self.synchronise
|
||||
FileUtils.mkdir_p(directory, mode: 0o700)
|
||||
File.open(directory.join('.registry.lock'), File::RDWR | File::CREAT, 0o600) do |lock|
|
||||
lock.flock(File::LOCK_EX)
|
||||
yield
|
||||
end
|
||||
end
|
||||
|
||||
def self.write(state)
|
||||
path = directory.join("#{ state.fetch('id') }.json")
|
||||
temporary = "#{ path }.#{ SecureRandom.hex(8) }.tmp"
|
||||
File.open(temporary, 'w', 0o600) do |file|
|
||||
file.write(JSON.generate(state))
|
||||
file.flush
|
||||
file.fsync
|
||||
end
|
||||
File.rename(temporary, path)
|
||||
ensure
|
||||
File.unlink(temporary) if temporary && File.exist?(temporary)
|
||||
end
|
||||
|
||||
def self.read(id)
|
||||
JSON.parse(File.read(directory.join("#{ id }.json")))
|
||||
rescue Errno::ENOENT
|
||||
nil
|
||||
end
|
||||
|
||||
def self.remove_download(state)
|
||||
name = state['download_file']
|
||||
return unless name&.match?(/\Acounts-[a-zA-Z0-9_-]+\.export\z/)
|
||||
|
||||
File.unlink(CountsExportDownload.directory.join(name))
|
||||
rescue Errno::ENOENT
|
||||
nil
|
||||
end
|
||||
|
||||
def self.remove_partial(id)
|
||||
Dir.glob(CountsExportDownload.directory.join("counts-#{ id }-*")).each do |path|
|
||||
File.unlink(path)
|
||||
rescue Errno::ENOENT
|
||||
nil
|
||||
end
|
||||
end
|
||||
|
||||
def self.clean
|
||||
Dir.glob(directory.join('*.tmp')).each do |path|
|
||||
File.unlink(path) if File.mtime(path) < Time.current - CountsExportDownload::EXPIRY
|
||||
end
|
||||
Dir.glob(directory.join('*.run')).each do |path|
|
||||
id = File.basename(path, '.run')
|
||||
next if directory.join("#{ id }.json").exist?
|
||||
|
||||
File.open(path, File::RDWR) do |lock|
|
||||
File.unlink(path) if lock.flock(File::LOCK_EX | File::LOCK_NB)
|
||||
end
|
||||
end
|
||||
Dir.glob(directory.join('*.json')).each do |path|
|
||||
state = JSON.parse(File.read(path))
|
||||
next if state.fetch('expires_at') > Time.current.to_f
|
||||
|
||||
run = directory.join("#{ state.fetch('id') }.run")
|
||||
File.open(run, File::RDWR | File::CREAT, 0o600) do |lock|
|
||||
unless lock.flock(File::LOCK_EX | File::LOCK_NB)
|
||||
state['status'] = 'cancelling'
|
||||
write(state)
|
||||
next
|
||||
end
|
||||
remove_download(state)
|
||||
remove_partial(state.fetch('id'))
|
||||
File.unlink(path)
|
||||
File.unlink(run)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def self.start(payload, owner, language)
|
||||
payload = payload.dup
|
||||
time_key = payload.key?('definition') ? 'execution_at' : 'at'
|
||||
if payload[time_key].nil? || payload[time_key] == ''
|
||||
payload[time_key] = Time.current.utc.iso8601(6)
|
||||
end
|
||||
exporter = CountsExporter.new(payload, language_code: language)
|
||||
authority = HistoryAccess.authority
|
||||
synchronise do
|
||||
clean
|
||||
raise WorkerUnavailable unless worker_available?
|
||||
active = Dir.glob(directory.join('*.json')).count do |path|
|
||||
existing = JSON.parse(File.read(path))
|
||||
expire_stalled(existing)
|
||||
ACTIVE.include?(existing['status'])
|
||||
end
|
||||
raise CountsExportDownload::Busy if active >= CountsExportDownload::MAX_CONCURRENT_EXPORTS
|
||||
|
||||
id = SecureRandom.hex(24)
|
||||
state = { 'id' => id, 'owner' => owner, 'payload' => payload,
|
||||
'language' => language, 'resource' => exporter.resource,
|
||||
'format' => exporter.format, 'status' => 'queued',
|
||||
'phase' => 'queued', 'created_at' => Time.current.to_f,
|
||||
'updated_at' => Time.current.to_f, 'heartbeat_at' => Time.current.to_f,
|
||||
'processed' => 0, 'total' => nil,
|
||||
**authority,
|
||||
'expires_at' => (Time.current + CountsExportDownload::EXPIRY).to_f }
|
||||
write(state)
|
||||
verifier.generate({ 'id' => id, 'owner' => owner },
|
||||
expires_in: CountsExportDownload::EXPIRY, purpose: 'counts_export_job')
|
||||
end
|
||||
end
|
||||
|
||||
def self.authorised_id(token, owner)
|
||||
value = verifier.verified(token, purpose: 'counts_export_job')
|
||||
return unless value.is_a?(Hash) && value['owner'] == owner
|
||||
return unless value['id']&.match?(/\A[a-f0-9]{48}\z/)
|
||||
|
||||
value['id']
|
||||
end
|
||||
|
||||
def self.status(token, owner, cancel: false)
|
||||
id = authorised_id(token, owner)
|
||||
return unless id
|
||||
snapshot = read(id)
|
||||
return unless snapshot && snapshot['owner'] == owner
|
||||
return unless HistoryAccess.authorised_artifact?(snapshot)
|
||||
|
||||
synchronise do
|
||||
state = read(id)
|
||||
return unless state && state['owner'] == owner && state['expires_at'] > Time.current.to_f
|
||||
|
||||
expire_stalled(state)
|
||||
|
||||
if cancel && ['queued', 'running', 'ready'].include?(state['status'])
|
||||
state['status'] = state['status'] == 'running' ? 'cancelling' : 'cancelled'
|
||||
remove_download(state)
|
||||
state.delete('token')
|
||||
write(state)
|
||||
end
|
||||
state.slice('status', 'phase', 'processed', 'total', 'format', 'token', 'error',
|
||||
'updated_at', 'heartbeat_at', 'download_attempt', 'download_status')
|
||||
end
|
||||
end
|
||||
|
||||
def self.download_attempt(token, owner, file_token: nil, attempt: nil, status: nil)
|
||||
id = authorised_id(token, owner)
|
||||
return unless id
|
||||
|
||||
synchronise do
|
||||
state = read(id)
|
||||
return unless (state && state['owner'] == owner && state['status'] == 'ready' &&
|
||||
state['expires_at'] > Time.current.to_f && HistoryAccess.authorised_artifact?(state))
|
||||
if attempt
|
||||
return unless state['download_attempt'] == attempt && state['token'] == file_token
|
||||
return unless state['download_status'] == 'pending'
|
||||
|
||||
state['download_status'] = status if status
|
||||
else
|
||||
state['download_attempt'] = SecureRandom.hex(24)
|
||||
state['download_status'] = 'pending'
|
||||
end
|
||||
write(state)
|
||||
state.slice('download_attempt', 'download_status', 'token')
|
||||
end
|
||||
end
|
||||
|
||||
def self.checkpoint(id, processed = nil, total = nil, phase: nil)
|
||||
synchronise do
|
||||
state = read(id)
|
||||
unless state && state['status'] == 'running' && state['expires_at'] > Time.current.to_f
|
||||
raise Cancelled
|
||||
end
|
||||
if processed || phase
|
||||
state.merge!('processed' => processed, 'total' => total) unless processed.nil?
|
||||
state['phase'] = phase if phase
|
||||
state['updated_at'] = Time.current.to_f
|
||||
write(state)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def self.work_one
|
||||
pulse
|
||||
FileUtils.mkdir_p(directory, mode: 0o700)
|
||||
Dir.glob(directory.join('*.json')).sort.each do |path|
|
||||
id = File.basename(path, '.json')
|
||||
File.open(directory.join("#{ id }.run"), File::RDWR | File::CREAT, 0o600) do |lock|
|
||||
next unless lock.flock(File::LOCK_EX | File::LOCK_NB)
|
||||
|
||||
state = synchronise do
|
||||
candidate = read(id)
|
||||
next unless candidate && ACTIVE.include?(candidate['status'])
|
||||
|
||||
if candidate['status'] != 'queued'
|
||||
candidate['status'] = candidate['status'] == 'cancelling' ? 'cancelled' : 'failed'
|
||||
remove_partial(id)
|
||||
candidate['error'] = 'interrupted'
|
||||
write(candidate)
|
||||
next
|
||||
end
|
||||
candidate['status'] = 'running'
|
||||
candidate['phase'] = 'aggregating'
|
||||
candidate['heartbeat_at'] = Time.current.to_f
|
||||
candidate['updated_at'] = Time.current.to_f
|
||||
write(candidate)
|
||||
candidate
|
||||
end
|
||||
next unless state
|
||||
|
||||
supervise(state)
|
||||
return true
|
||||
end
|
||||
end
|
||||
synchronise { clean }
|
||||
false
|
||||
end
|
||||
|
||||
def self.supervise(state)
|
||||
id = state.fetch('id')
|
||||
ApplicationRecord.connection_handler.clear_all_connections!
|
||||
pid = Process.fork do
|
||||
Signal.trap('TERM') { raise Cancelled }
|
||||
Signal.trap('INT') { raise Cancelled }
|
||||
begin
|
||||
perform(state)
|
||||
ensure
|
||||
ApplicationRecord.connection_handler.clear_all_connections!
|
||||
Process.exit!(0)
|
||||
end
|
||||
end
|
||||
started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
||||
termination = nil
|
||||
reaped = false
|
||||
begin
|
||||
loop do
|
||||
if Process.waitpid(pid, Process::WNOHANG)
|
||||
reaped = true
|
||||
break
|
||||
end
|
||||
pulse
|
||||
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
||||
terminal = synchronise do
|
||||
current = read(id)
|
||||
if current && current['status'] == 'running'
|
||||
if now - started > CountsExportDownload::MAX_EXPORT_SECONDS
|
||||
current.merge!('status' => 'failed', 'error' => 'generation_timeout')
|
||||
end
|
||||
current['heartbeat_at'] = Time.current.to_f
|
||||
write(current)
|
||||
end
|
||||
!current || !['running', 'ready'].include?(current['status'])
|
||||
end
|
||||
begin
|
||||
if terminal && termination.nil?
|
||||
Process.kill('TERM', pid)
|
||||
termination = now
|
||||
elsif termination && now - termination > 5
|
||||
Process.kill('KILL', pid)
|
||||
end
|
||||
rescue Errno::ESRCH
|
||||
# The child may exit between the nonblocking wait and signal delivery.
|
||||
nil
|
||||
end
|
||||
sleep(1)
|
||||
end
|
||||
ensure
|
||||
unless reaped
|
||||
begin
|
||||
Process.kill('KILL', pid)
|
||||
rescue Errno::ESRCH
|
||||
nil
|
||||
end
|
||||
Process.waitpid(pid)
|
||||
end
|
||||
synchronise do
|
||||
current = read(id)
|
||||
if current && current['status'] != 'ready'
|
||||
if ['running', 'cancelling'].include?(current['status'])
|
||||
current['status'] = current['status'] == 'cancelling' ? 'cancelled' : 'failed'
|
||||
current['error'] = 'interrupted' if current['status'] == 'failed'
|
||||
end
|
||||
remove_download(current)
|
||||
remove_partial(id)
|
||||
current.delete('token')
|
||||
current['updated_at'] = Time.current.to_f
|
||||
write(current)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def self.perform(state)
|
||||
actor = state['actor_id'] && User.find_by(id: state['actor_id'])
|
||||
actor = nil unless state['admin'] && actor&.admin? && !actor.banned?
|
||||
HistoryAccess.set(user: actor) do
|
||||
unless HistoryAccess.authorised_artifact?(state)
|
||||
synchronise do
|
||||
current = read(state.fetch('id'))
|
||||
current['status'] = 'failed'
|
||||
write(current)
|
||||
end
|
||||
return
|
||||
end
|
||||
perform_authorised(state)
|
||||
end
|
||||
end
|
||||
|
||||
def self.perform_authorised(state)
|
||||
id = state.fetch('id')
|
||||
token = nil
|
||||
begin
|
||||
checkpoint(id)
|
||||
exporter = CountsExporter.new(state.fetch('payload'), language_code: state.fetch('language'))
|
||||
last = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
||||
progress = lambda do |processed, total|
|
||||
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
||||
if processed.zero? || now - last >= 0.25 || processed == total
|
||||
checkpoint(id, processed, total)
|
||||
last = now
|
||||
end
|
||||
end
|
||||
check = lambda do
|
||||
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
||||
if now - last >= 0.25
|
||||
checkpoint(id)
|
||||
last = now
|
||||
end
|
||||
end
|
||||
phase = ->(value) { checkpoint(id, phase: value) }
|
||||
token = CountsExportDownload.prepare(exporter, state.fetch('owner'),
|
||||
progress:, phase:, check:, identifier: id)
|
||||
file = CountsExportDownload.find(token, state.fetch('owner'))
|
||||
raise Cancelled unless file
|
||||
|
||||
synchronise do
|
||||
current = read(id)
|
||||
if (!current || current['status'] != 'running' ||
|
||||
current['expires_at'] <= Time.current.to_f)
|
||||
File.unlink(file.fetch('path')) if File.exist?(file.fetch('path'))
|
||||
raise Cancelled
|
||||
end
|
||||
current.merge!('status' => 'ready', 'token' => token,
|
||||
'updated_at' => Time.current.to_f,
|
||||
'download_file' => File.basename(file.fetch('path')))
|
||||
current['total'] = current['processed']
|
||||
write(current)
|
||||
end
|
||||
rescue StandardError => error
|
||||
remove_partial(id)
|
||||
synchronise do
|
||||
current = read(id)
|
||||
if current
|
||||
unless current['status'] == 'failed'
|
||||
current['status'] = error.is_a?(Cancelled) ? 'cancelled' : 'failed'
|
||||
current['error'] = error.class.name
|
||||
end
|
||||
current['updated_at'] = Time.current.to_f
|
||||
write(current)
|
||||
end
|
||||
end
|
||||
unless error.is_a?(Cancelled)
|
||||
Rails.logger.error("counts_export_job #{ id }: #{ error.class }")
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,286 @@
|
||||
require 'set'
|
||||
|
||||
# Export identifiers are stable; UI translations are supplied separately as column labels.
|
||||
class CountsExportProjection
|
||||
ROOT_KEYS = { 'event_type' => 'event_type', 'resource_id' => 'resource_id', 'version_no' => 'version_no',
|
||||
'latest' => 'latest', 'thumbnail' => 'thumbnail_url',
|
||||
'thumbnail_base' => 'thumbnail_base', 'title' => 'title', 'url' => 'url',
|
||||
'tags' => 'tags', 'parents' => 'parents', 'video_ms' => 'video_ms',
|
||||
'original_created_from' => 'original_created_from',
|
||||
'original_created_before' => 'original_created_before',
|
||||
'user' => 'created_by_user', 'created_at' => 'created_at',
|
||||
'name' => 'name', 'names' => 'names', 'category' => 'category',
|
||||
'aliases' => 'aliases', 'deprecated' => 'deprecated' }.freeze
|
||||
CHILD_KEYS = { 'id' => 'id', 'versionNo' => 'version_no', 'kind' => 'kind',
|
||||
'name' => 'name', 'category' => 'category', 'sections' => 'sections',
|
||||
'names' => 'names', 'aliases' => 'aliases',
|
||||
'languageCode' => 'language', 'scriptCode' => 'script',
|
||||
'autoGenerated' => 'auto_generated' }.freeze
|
||||
NAME_FIELDS = ['languageCode', 'scriptCode', 'name', 'aliases', 'autoGenerated'].freeze
|
||||
CHILDREN = {
|
||||
'tags' => ['id', 'versionNo', 'kind', 'name', 'category', 'sections', 'names', 'aliases'],
|
||||
'names' => NAME_FIELDS,
|
||||
'aliases' => ['languageCode', 'name', 'scriptCode'],
|
||||
'parents' => ['id', 'name', 'names'] }.freeze
|
||||
|
||||
def self.default_columns(resource)
|
||||
fields = { 'posts' => ['title', 'url', 'tags', 'parents', 'video_ms'],
|
||||
'tags' => ['name', 'category', 'aliases', 'parents'],
|
||||
'nico_tags' => ['name', 'linked_tags'],
|
||||
'tag_descriptions' => ['language_code', 'body', 'source_language_code'],
|
||||
'tag_groups' => ['name', 'description', 'members'],
|
||||
'materials' => ['url', 'tag_id', 'source_kind'] }.fetch(resource)
|
||||
['resource_id', 'version_no'] + fields + ['event_type', 'user', 'created_at']
|
||||
end
|
||||
|
||||
def self.default_children(resource)
|
||||
{ 'posts' => ['tags.name', 'parents.id'],
|
||||
'tags' => ['aliases.name', 'parents.id'],
|
||||
'tag_groups' => ['members.tagId'] }.fetch(resource, [])
|
||||
end
|
||||
|
||||
attr_reader :columns, :headers
|
||||
|
||||
def initialize(params, resource:, summary:, tags:)
|
||||
params = CountsResourceDefinition.normalise_params(params)
|
||||
if params.key?('definition')
|
||||
params = CountsDefaults.for(params.merge('resource' => resource)).merge(params)
|
||||
end
|
||||
@params = params
|
||||
definition = CountsResourceDefinition.new(resource)
|
||||
name_keys = definition.name_columns.map { _1[:key] }
|
||||
@root_keys = ROOT_KEYS.merge(name_keys.to_h { [_1, _1] })
|
||||
@child_keys = CHILD_KEYS.merge(name_keys.to_h { [_1, _1] })
|
||||
@children = CHILDREN.transform_values(&:dup)
|
||||
['tags', 'parents'].each { @children[_1] += name_keys }
|
||||
@complete_names = params.fetch('format', 'json') == 'json'
|
||||
@resource = resource
|
||||
@labels = JSON.parse(params.fetch('labels', '{}'))
|
||||
raise ArgumentError unless @labels.is_a?(Hash) && @labels.length <= 100
|
||||
raise ArgumentError unless @labels.all? { |key, value| valid_label?(key, value) }
|
||||
@summary = summary
|
||||
allowed = if summary
|
||||
tags ? ['kind', 'tag_id', 'name', 'deprecated', 'count'] : ['group', 'count']
|
||||
else
|
||||
CountsQuery::COLUMNS.fetch(resource) + (resource == 'tags' ? name_keys : [])
|
||||
end
|
||||
defaults = summary ? allowed : self.class.default_columns(resource)
|
||||
@columns = QueryState.list(params.fetch('columns', defaults.join(' ')))
|
||||
raise ArgumentError if (@columns - allowed).any? || @columns.uniq != @columns
|
||||
|
||||
@headers = JSON.parse(params.fetch('headers', JSON.generate(@columns)))
|
||||
unless (@headers.is_a?(Array) && @headers.length == @columns.length &&
|
||||
@headers.all? { _1.is_a?(String) && _1.length <= 200 })
|
||||
raise ArgumentError
|
||||
end
|
||||
@order = QueryState.list(params.fetch('order', ''))
|
||||
@sorts = QueryState.list(params.fetch('child_sort', '')).map do |entry|
|
||||
parts = entry.split(':', -1)
|
||||
raise ArgumentError unless parts.length == 2
|
||||
|
||||
path, direction = parts
|
||||
raise ArgumentError unless ['asc', 'desc'].include?(direction)
|
||||
|
||||
[path, direction]
|
||||
end
|
||||
@selected = params.key?('children') ? QueryState.list(params['children']).to_set :
|
||||
self.class.default_children(resource).to_set
|
||||
valid = @children.flat_map do |key, fields|
|
||||
paths(fields, key)
|
||||
end
|
||||
raise ArgumentError if @selected && (@selected.to_a - valid).any?
|
||||
prune = lambda do |fields, prefix|
|
||||
fields.each do |key|
|
||||
path = "#{ prefix }.#{ key }"
|
||||
children = fields_for(key)
|
||||
next if children.empty?
|
||||
|
||||
prune.call(children, path)
|
||||
@selected.delete(path) unless children.any? { @selected.include?("#{ path }.#{ _1 }") }
|
||||
end
|
||||
end
|
||||
@children.each { |key, fields| prune.call(fields, key) }
|
||||
pairs = @columns.zip(@headers).select do |key, _header|
|
||||
!@children.key?(key) || @children[key].any? { @selected.include?("#{ key }.#{ _1 }") }
|
||||
end
|
||||
@columns = pairs.map(&:first)
|
||||
@headers = pairs.map(&:last)
|
||||
allowed_sorts = valid.map do |path|
|
||||
parts = path.split('.')
|
||||
field = parts.pop
|
||||
"#{ parts.join('.') }/#{ field }"
|
||||
end
|
||||
if ((@sorts.map(&:first) - allowed_sorts).any? ||
|
||||
@sorts.map(&:first).uniq.length != @sorts.length)
|
||||
raise ArgumentError
|
||||
end
|
||||
end
|
||||
|
||||
def metadata
|
||||
@columns.each_with_index.map do |key, index|
|
||||
{ key: @summary ? key : @root_keys.fetch(key), label: @headers[index] }
|
||||
end
|
||||
end
|
||||
|
||||
def project(row)
|
||||
result = @columns.to_h do |key|
|
||||
if @summary
|
||||
value = key == 'group' ? { key: row[:key], name: row[:name] } : row[key.to_sym]
|
||||
[key, value]
|
||||
else
|
||||
value = case key
|
||||
when 'thumbnail' then row[:thumbnail_base]
|
||||
else row[@root_keys.fetch(key).to_sym]
|
||||
end
|
||||
value = nested(value || [], @children.fetch(key), key) if @children.key?(key)
|
||||
[@root_keys.fetch(key), value]
|
||||
end
|
||||
end
|
||||
if @complete_names && ((!@summary && @resource == 'tags') || row.key?(:names))
|
||||
result['names'] = row[:names] || []
|
||||
end
|
||||
result['aliases'] = row[:aliases] if @complete_names && row.key?(:aliases)
|
||||
result
|
||||
end
|
||||
|
||||
def csv_record(record, row)
|
||||
record.map do |key, value|
|
||||
key == 'group' ? row[:name] || row[:key] : csv_display(value, key)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def valid_label?(key, value)
|
||||
key.is_a?(String) && value.is_a?(String) && key.length <= 100 && value.length <= 300
|
||||
end
|
||||
|
||||
def csv_display(value, key)
|
||||
if key.start_with?('aliases_') && value.is_a?(Array)
|
||||
return value.map { fetch(_1, 'name') }.compact.join("\n")
|
||||
end
|
||||
return @labels.fetch('bot', '') if key == 'created_by_user' && value.nil?
|
||||
return @labels.fetch("event_type:#{ value }", value) if key == 'event_type'
|
||||
return value ? @labels.fetch('latest', 'true') : '' if key == 'latest'
|
||||
return @labels.fetch("kind:#{ value }", value) if key == 'kind'
|
||||
return @labels.fetch("category:#{ value }", value) if key == 'category'
|
||||
if ['language', 'script'].include?(key) && value.is_a?(Hash)
|
||||
return @labels.fetch("language:#{ value[:code] }", value[:code]) if key == 'language'
|
||||
return @labels.fetch("#{ key }:#{ value[:code] }",
|
||||
value[:name] || @labels.fetch('notRegistered', ''))
|
||||
end
|
||||
return @labels.fetch("boolean:#{ value }", value) if value == true || value == false
|
||||
return value.map { csv_display(_1, key) } if value.is_a?(Array)
|
||||
if value.is_a?(Hash)
|
||||
return value.to_h do |field, item|
|
||||
[@labels.fetch(field.to_s, field), csv_display(item, field.to_s)]
|
||||
end
|
||||
end
|
||||
|
||||
value
|
||||
end
|
||||
|
||||
def fields_for(key)
|
||||
return NAME_FIELDS if key == 'names'
|
||||
return ['languageCode', 'name', 'scriptCode'] if key == 'aliases'
|
||||
|
||||
[]
|
||||
end
|
||||
|
||||
def paths(fields, prefix)
|
||||
fields.flat_map do |key|
|
||||
path = "#{ prefix }.#{ key }"
|
||||
[path, *paths(fields_for(key), path)]
|
||||
end
|
||||
end
|
||||
|
||||
def fetch(row, key) = row.key?(key) ? row[key] : row[key.to_sym]
|
||||
|
||||
def named_code(row, kind)
|
||||
code = fetch(row, "#{ kind }_code")
|
||||
name = fetch(row, "#{ kind }_name")
|
||||
{ code:, name: name.present? && name.to_s.downcase != code.to_s.downcase ? name : nil }
|
||||
end
|
||||
|
||||
def stable(value)
|
||||
case value
|
||||
when Hash
|
||||
value.stringify_keys.sort.to_h.transform_values { stable(_1) }
|
||||
when Array
|
||||
value.map { stable(_1) }
|
||||
else
|
||||
value
|
||||
end
|
||||
end
|
||||
|
||||
def sort_value(row, key)
|
||||
return fetch(row, 'language_code') if key == 'languageCode'
|
||||
return fetch(row, 'script_code') if key == 'scriptCode'
|
||||
|
||||
fetch(row, @child_keys.fetch(key))
|
||||
end
|
||||
|
||||
def comparison(left, right)
|
||||
if left.is_a?(Array) && right.is_a?(Array) && left.length != right.length
|
||||
return left.length <=> right.length
|
||||
end
|
||||
return left <=> right if left.is_a?(Numeric) && right.is_a?(Numeric)
|
||||
if [true, false].include?(left) && [true, false].include?(right)
|
||||
return (left ? 1 : 0) <=> (right ? 1 : 0)
|
||||
end
|
||||
|
||||
a = left.is_a?(Hash) || left.is_a?(Array) ? JSON.generate(stable(left)) : left.to_s
|
||||
b = right.is_a?(Hash) || right.is_a?(Array) ? JSON.generate(stable(right)) : right.to_s
|
||||
a <=> b
|
||||
end
|
||||
|
||||
def sorted(rows, prefix, fields)
|
||||
sorts = @sorts.filter_map do |path, direction|
|
||||
parent, key = path.split('/', 2)
|
||||
next unless parent == prefix
|
||||
raise ArgumentError unless fields.include?(key)
|
||||
|
||||
[key, direction]
|
||||
end
|
||||
rows.each_with_index.sort do |(left, a), (right, b)|
|
||||
result = 0
|
||||
sorts.each do |key, direction|
|
||||
first = sort_value(left, key)
|
||||
last = sort_value(right, key)
|
||||
result = if first.nil? || last.nil?
|
||||
first.nil? == last.nil? ? 0 : (first.nil? ? 1 : -1)
|
||||
else
|
||||
comparison(first, last) * (direction == 'asc' ? 1 : -1)
|
||||
end
|
||||
break unless result.zero?
|
||||
end
|
||||
result.zero? ? a <=> b : result
|
||||
end.map(&:first)
|
||||
end
|
||||
|
||||
def nested(rows, fields, prefix)
|
||||
keys = fields.select { !@selected || @selected.include?("#{ prefix }.#{ _1 }") }
|
||||
ordered = (@order + keys.map { "#{ prefix }.#{ _1 }" }).uniq
|
||||
.filter_map { |path| keys.find { "#{ prefix }.#{ _1 }" == path } }
|
||||
sorted(rows, prefix, fields).map do |row|
|
||||
result = ordered.to_h do |key|
|
||||
stable = @child_keys.fetch(key)
|
||||
value = case key
|
||||
when 'languageCode' then named_code(row, 'language')
|
||||
when 'scriptCode' then named_code(row, 'script')
|
||||
else fetch(row, stable)
|
||||
end
|
||||
if fields_for(key).any?
|
||||
value = nested(value || [], fields_for(key), "#{ prefix }.#{ key }")
|
||||
end
|
||||
[stable, value]
|
||||
end
|
||||
if @complete_names && ['tags', 'parents'].include?(prefix)
|
||||
result['names'] = fetch(row, 'names') || []
|
||||
aliases = fetch(row, 'aliases')
|
||||
result['aliases'] = aliases if aliases
|
||||
end
|
||||
result
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,157 @@
|
||||
require 'csv'
|
||||
|
||||
# One read view and a fixed point for the entire download; only one page is held in memory.
|
||||
class CountsExporter
|
||||
EXTRA_KEYS = ['basis', 'format', 'resource', 'headers', 'children', 'order',
|
||||
'summary_columns', 'nested_json', 'child_sort', 'labels'].freeze
|
||||
attr_reader :format, :resource
|
||||
|
||||
def initialize(params, language_code:)
|
||||
params = CountsResourceDefinition.normalise_params(params)
|
||||
if params.key?('definition')
|
||||
@table_export = CountsTableExport.new(params)
|
||||
@format = @table_export.format
|
||||
@resource = @table_export.resource
|
||||
return
|
||||
end
|
||||
allowed = (CountsQuery::KEYS + CountsStateQuery::KEYS + EXTRA_KEYS).uniq
|
||||
QueryState.validate!(params, allowed)
|
||||
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
|
||||
|
||||
@params = params
|
||||
@language = language_code
|
||||
@format = params.fetch('format', 'json')
|
||||
@resource = params.fetch('resource', 'posts')
|
||||
@mode = params.fetch('mode', 'summary')
|
||||
@history = params.fetch('basis', 'state') == 'history'
|
||||
raise ArgumentError unless ['json', 'csv'].include?(format)
|
||||
raise ArgumentError unless ['posts', 'tags'].include?(resource)
|
||||
raise ArgumentError unless ['summary', 'details'].include?(@mode)
|
||||
raise ArgumentError unless ['state', 'history'].include?(params.fetch('basis', 'state'))
|
||||
|
||||
@query_params = params.except(*EXTRA_KEYS, 'columns', 'page', 'limit')
|
||||
.merge('mode' => @mode, 'page' => '1', 'limit' => '100')
|
||||
point = CountsStateQuery.new(@query_params.slice('at').merge('resource' => resource),
|
||||
language_code: @language).at
|
||||
@query_params['at'] = point.iso8601(6)
|
||||
@query_params['axis'] ||= !@history && resource == 'posts' ? 'tags' : 'month'
|
||||
@projection = CountsExportProjection.new(params, resource:,
|
||||
summary: @mode == 'summary',
|
||||
tags: !@history && @query_params['axis'] == 'tags')
|
||||
if @history
|
||||
@query_params.delete('kind')
|
||||
history_query
|
||||
else
|
||||
@query_params.delete('latest')
|
||||
@state_query = CountsStateQuery.new(@query_params.merge('resource' => resource),
|
||||
language_code: @language).validate_query!
|
||||
end
|
||||
end
|
||||
|
||||
def body(progress: nil, phase: nil)
|
||||
return @table_export.body(progress:, phase:) if @table_export
|
||||
|
||||
Enumerator.new do |output|
|
||||
ApplicationRecord.transaction(isolation: :repeatable_read) do
|
||||
processed = 0
|
||||
first_row = true
|
||||
started = false
|
||||
phase&.call('aggregating')
|
||||
pages do |page|
|
||||
total = @mode == 'details' ? page[:count] : nil
|
||||
progress&.call(processed, total)
|
||||
phase&.call('writing')
|
||||
unless started
|
||||
meta = { resource:, mode: @mode, basis: @history ? 'history' : 'state',
|
||||
at: @query_params['at'], boundary: 'inclusive',
|
||||
filters: @query_params.except('sort', 'page', 'limit', 'mode'),
|
||||
sort: export_sort,
|
||||
child_sort: sort_metadata(@params.fetch('child_sort', '')),
|
||||
columns: @projection.metadata,
|
||||
children: @params['children']&.split(' '),
|
||||
field_order: QueryState.list(@params.fetch('order', '')), timezone: 'UTC',
|
||||
count: @mode == 'summary' ? page[:group_count] : page[:count],
|
||||
totals: page[:totals], coverage: page[:coverage] }
|
||||
output << (format == 'json' ? "{\"metadata\":#{ meta.to_json },\"data\":[" : "\uFEFF")
|
||||
if format == 'csv'
|
||||
output << CSV.generate_line(
|
||||
@projection.headers.map { csv_value(_1) }, row_sep: "\r\n")
|
||||
end
|
||||
started = true
|
||||
end
|
||||
(page[:versions] || page[:state_rows] || page[:groups] || []).each do |row|
|
||||
record = @projection.project(row)
|
||||
if format == 'json'
|
||||
output << ',' unless first_row
|
||||
output << record.to_json
|
||||
else
|
||||
values = @projection.csv_record(record, row).map { csv_value(_1) }
|
||||
output << CSV.generate_line(values, row_sep: "\r\n")
|
||||
end
|
||||
first_row = false
|
||||
processed += 1
|
||||
progress&.call(processed, total)
|
||||
end
|
||||
phase&.call('aggregating')
|
||||
end
|
||||
progress&.call(processed, processed)
|
||||
output << ']}' if format == 'json'
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def export_sort
|
||||
sort_metadata(@query_params.fetch('sort', default_sort))
|
||||
end
|
||||
|
||||
def sort_metadata(value)
|
||||
value.split(' ').map do |entry|
|
||||
key, direction = entry.split(':', 2)
|
||||
{ key:, direction: }
|
||||
end
|
||||
end
|
||||
|
||||
def default_sort
|
||||
return 'created_at:desc' if @mode == 'details'
|
||||
|
||||
!@history && @query_params['axis'] == 'tags' ? 'count:desc' : 'group:asc'
|
||||
end
|
||||
|
||||
def history_query
|
||||
scope = resource == 'posts' ? PostVersion.with_previous_version : TagVersion.all
|
||||
CountsQuery.new(scope, resource, @query_params)
|
||||
end
|
||||
|
||||
def pages
|
||||
unless @history
|
||||
@state_query.each_page { yield _1 }
|
||||
return
|
||||
end
|
||||
query = history_query
|
||||
total = @mode == 'summary' ? nil : query.count
|
||||
loop do
|
||||
result = if @mode == 'summary'
|
||||
query.summary(include_count: @query_params['page'] == '1')
|
||||
else
|
||||
{ count: total,
|
||||
versions: CountsVersionRepr.many(query.details, resource:,
|
||||
language_code: @language,
|
||||
at: Time.iso8601(@query_params['at']),
|
||||
history_flg: true) }
|
||||
end
|
||||
total ||= result[:group_count]
|
||||
yield result
|
||||
break if @query_params['page'].to_i * 100 >= total
|
||||
|
||||
@query_params['page'] = (@query_params['page'].to_i + 1).to_s
|
||||
query = history_query
|
||||
end
|
||||
end
|
||||
|
||||
def csv_value(value)
|
||||
text = value.is_a?(Hash) || value.is_a?(Array) ? JSON.generate(value) : value.to_s
|
||||
text.match?(/\A\s*[=+@-]/) ? "'#{ text }" : text
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,51 @@
|
||||
# Coordinates carry their aggregation depth; SQL NULL remains an ordinary axis value.
|
||||
class CountsPivotLayout
|
||||
def self.depths(keys, sorts, measures: true)
|
||||
return [] if keys.length < 2 || sorts.first&.first == 'count' ||
|
||||
sorts.first&.first&.match?(/\Am[0-9]+\z/)
|
||||
|
||||
relevant = sorts.map(&:first).select { keys.include?(_1) || (measures && (_1 == 'count' ||
|
||||
_1.match?(/\Am[0-9]+\z/))) }
|
||||
order = (relevant + keys).uniq
|
||||
(1...keys.length).select { order.first(_1).sort == keys.first(_1).sort }
|
||||
end
|
||||
|
||||
def self.label_key(row_count, column_count, row_depth, column_depth)
|
||||
total = row_count + column_count
|
||||
collapsed = total - row_depth - column_depth
|
||||
return nil if collapsed.zero?
|
||||
return 'grandTotal' if total >= 3 && collapsed == total
|
||||
if (collapsed == 1 &&
|
||||
((row_count > 1 && row_depth < row_count) ||
|
||||
(column_count > 1 && column_depth < column_count)))
|
||||
return 'subtotal'
|
||||
end
|
||||
'singleTotal'
|
||||
end
|
||||
|
||||
def self.headers(values, depths, next_values: nil, grand: false, axis_count: nil,
|
||||
other_count: 0, direction: :row)
|
||||
result = []
|
||||
values.each_with_index do |tuple, index|
|
||||
result << { values: tuple, kind: 'data' }
|
||||
following = values[index + 1] || next_values
|
||||
depths.reverse_each do |depth|
|
||||
prefix = tuple.first(depth)
|
||||
next if following && following.first(depth) == prefix
|
||||
|
||||
result << { values: prefix, kind: 'subtotal' }
|
||||
end
|
||||
end
|
||||
result << { values: [], kind: 'grand_total' } if grand
|
||||
count = axis_count || values.first&.length || 0
|
||||
result.each do |header|
|
||||
depth = header[:values].length
|
||||
label = if header[:kind] != 'data'
|
||||
direction == :row ? label_key(count, other_count, depth, other_count) :
|
||||
label_key(other_count, count, other_count, depth)
|
||||
end
|
||||
header.merge!(depth:, axis_count: count, label_key: label)
|
||||
end
|
||||
result
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,82 @@
|
||||
# Re-aggregate the population, never add up already grouped cells (multi-value axes overlap).
|
||||
class CountsPivotTotals
|
||||
def initialize(population, axes, row_headers, column_headers)
|
||||
@population = population
|
||||
@axes = axes
|
||||
@rows = row_headers.group_by { _1[:values].length }
|
||||
@columns = column_headers.group_by { _1[:values].length }
|
||||
end
|
||||
|
||||
def call
|
||||
# MySQL's unlimited LIMIT prevents CTE merging: authorise once for all branches.
|
||||
basis = @population.scope.except(:select, :order).select('v.*')
|
||||
.limit(18_446_744_073_709_551_615).to_sql
|
||||
queries = @rows.flat_map do |row_depth, headers|
|
||||
@columns.filter_map do |column_depth, columns|
|
||||
next if row_depth == @axes.rows.length && column_depth == @axes.columns.length
|
||||
|
||||
indexes = (0...row_depth).to_a +
|
||||
(0...column_depth).map { @axes.rows.length + _1 }
|
||||
expressions = indexes.map { @axes.expressions[_1] }
|
||||
scope = @population.scope.except(:select, :order, :where, :from).from('authorised v')
|
||||
additive = @axes.rows.any? { _1['field'] == 'tags' }
|
||||
retained = additive ? (0...@axes.rows.length).to_a | indexes : indexes
|
||||
joins = @axes.joins(retained.map { @axes.all[_1].fetch('field') })
|
||||
scope = scope.joins(joins) unless joins.empty?
|
||||
if row_depth.positive?
|
||||
tuples = headers.map { _1[:values] }.uniq
|
||||
predicate = tuples.map { Array.new(row_depth) { |i|
|
||||
"(#{ expressions[i] }) <=> ?"
|
||||
}.join(' AND ') }.map { "(#{ _1 })" }.join(' OR ')
|
||||
scope = scope.where(predicate, *tuples.flatten)
|
||||
end
|
||||
if column_depth.positive?
|
||||
tuples = columns.map { _1[:values] }.uniq
|
||||
predicate = tuples.map { Array.new(column_depth) { |i|
|
||||
"(#{ expressions[row_depth + i] }) <=> ?"
|
||||
}.join(' AND ') }.map { "(#{ _1 })" }.join(' OR ')
|
||||
scope = scope.where(predicate, *tuples.flatten)
|
||||
end
|
||||
grouped = expressions.each_with_index.map { |expression, index|
|
||||
"#{ expression } AS v#{ index }"
|
||||
}
|
||||
# Fixed-width result coordinates, independent of NULL and of translated labels.
|
||||
padding = (expressions.length...@axes.all.length).map { "NULL AS v#{ _1 }" }
|
||||
fields = ["#{ row_depth } AS row_depth", "#{ column_depth } AS column_depth"] +
|
||||
grouped + padding + ['COUNT(DISTINCT v.id) AS count']
|
||||
if additive
|
||||
# Sum full row tuples, not page cells; preserve DISTINCT inside each tuple.
|
||||
leaf = retained.map { @axes.expressions[_1] }
|
||||
selected = leaf.each_with_index.map { |expression, index| "#{ expression } AS k#{ index }" }
|
||||
inner = scope.group(*leaf.map { Arel.sql(_1) })
|
||||
.select(Arel.sql((selected + ['COUNT(DISTINCT v.id) AS count']).join(', '))).to_sql
|
||||
coordinates = indexes.map { "k#{ retained.index(_1) }" }
|
||||
projected = coordinates.each_with_index.map { |key, index| "#{ key } AS v#{ index }" }
|
||||
output = ["#{ row_depth } AS row_depth", "#{ column_depth } AS column_depth"] +
|
||||
projected + padding + ['SUM(count) AS count']
|
||||
grouping = coordinates.empty? ? '' : " GROUP BY #{ coordinates.join(', ') }"
|
||||
"SELECT #{ output.join(', ') } FROM (#{ inner }) leaf#{ grouping }"
|
||||
else
|
||||
scope = scope.group(*expressions.map { Arel.sql(_1) }) if expressions.any?
|
||||
scope.optimizer_hints('MAX_EXECUTION_TIME(10000)')
|
||||
.select(Arel.sql(fields.join(', '))).to_sql
|
||||
end
|
||||
end
|
||||
end
|
||||
return [] if queries.empty?
|
||||
|
||||
sql = queries.map { "(#{ _1 })" }.join(' UNION ALL ')
|
||||
@population.definition.model.connection.select_all(
|
||||
"WITH authorised AS (#{ basis }) " +
|
||||
"SELECT /*+ MAX_EXECUTION_TIME(10000) */ * FROM (#{ sql }) totals").map do |record|
|
||||
row_depth = record['row_depth'].to_i
|
||||
column_depth = record['column_depth'].to_i
|
||||
{ row: (0...row_depth).map { record["v#{ _1 }"] },
|
||||
column: (0...column_depth).map { record["v#{ row_depth + _1 }"] },
|
||||
row_depth:, column_depth:,
|
||||
label_key: CountsPivotLayout.label_key(@axes.rows.length, @axes.columns.length,
|
||||
row_depth, column_depth),
|
||||
count: record['count'].to_i }
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,224 @@
|
||||
# Population first, then filters. Previous versions are joined before period filtering.
|
||||
class CountsPopulation
|
||||
attr_reader :definition, :params, :at, :history, :scope, :timezone
|
||||
|
||||
def initialize(params, authorisation_flg: true)
|
||||
@authorisation_flg = authorisation_flg
|
||||
@params = params
|
||||
@definition = CountsResourceDefinition.new(params.fetch('resource', 'posts'))
|
||||
@history = params.fetch('definition', 'latest') == 'history'
|
||||
raise ArgumentError unless ['latest', 'history'].include?(params.fetch('definition', 'latest'))
|
||||
|
||||
@timezone = TZInfo::Timezone.get(params.fetch('timezone', 'UTC'))
|
||||
@at = timestamp(params['execution_at']) || Time.current.utc
|
||||
raise ArgumentError if at > Time.current + 1.second
|
||||
|
||||
@scope = definition.model.all
|
||||
make_population
|
||||
@scope = HistoryAccess.visible(@scope, 'v') if @authorisation_flg
|
||||
apply_filters
|
||||
rescue TZInfo::InvalidTimezoneIdentifier
|
||||
raise ArgumentError
|
||||
end
|
||||
|
||||
def timestamp(value, end_date: false)
|
||||
return nil if value.nil? || value == ''
|
||||
raise ArgumentError unless value.is_a?(String)
|
||||
|
||||
if value.match?(/\A\d{4}-\d{2}-\d{2}\z/)
|
||||
date = Date.iso8601(value)
|
||||
date += 1 if end_date
|
||||
timezone.local_to_utc(Time.utc(date.year, date.month, date.day))
|
||||
else
|
||||
raise ArgumentError unless value.match?(/(?:Z|[+-]\d{2}:\d{2})\z/)
|
||||
|
||||
time = Time.iso8601(value).utc
|
||||
raise ArgumentError unless time.year.between?(1900, 2100)
|
||||
|
||||
time
|
||||
end
|
||||
rescue TZInfo::AmbiguousTime, TZInfo::PeriodNotFound, Date::Error
|
||||
raise ArgumentError
|
||||
end
|
||||
|
||||
def latest_sql(row = 'v', cutoff = nil)
|
||||
cutoff ||= history ? at : timestamp(params['at']) || at
|
||||
keys = definition.keys.map { "newer.#{ _1 } = #{ row }.#{ _1 }" }.join(' AND ')
|
||||
bound = definition.model.connection.quote(cutoff)
|
||||
<<~SQL.squish
|
||||
NOT EXISTS (SELECT 1 FROM #{ definition.table } newer
|
||||
WHERE #{ keys } AND newer.version_no > #{ row }.version_no
|
||||
AND newer.#{ definition.time_column } <= #{ bound })
|
||||
SQL
|
||||
end
|
||||
|
||||
def changed_sql(key)
|
||||
column = definition.column(key)
|
||||
"(previous.id IS NOT NULL AND NOT (v.#{ column } <=> previous.#{ column }))"
|
||||
end
|
||||
|
||||
# Unrestricted predicates are used only for numerical totals, never record output.
|
||||
def access_sql(model, row)
|
||||
@authorisation_flg ? HistoryAccess.sql(model, row) : 'TRUE'
|
||||
end
|
||||
|
||||
def axis_condition(scope, entry)
|
||||
key = entry.fetch('field')
|
||||
raise ArgumentError unless definition.axes(history).include?(key)
|
||||
if entry.key?('grain')
|
||||
unless definition.fields[key] == :datetime && CountsAxes::GRAINS.include?(entry['grain'])
|
||||
raise ArgumentError
|
||||
end
|
||||
end
|
||||
if key == 'tags'
|
||||
scope = scope.where(snapshot_membership(entry.fetch('value')))
|
||||
elsif key == 'changed_field'
|
||||
raise ArgumentError unless history && definition.changes.include?(entry['value'])
|
||||
|
||||
scope = scope.where(Arel.sql(changed_sql(entry['value'])))
|
||||
elsif definition.fields[key] == :datetime && entry['grain']
|
||||
scope = scope.where(definition.expression(key) => nil) if entry['value'].nil?
|
||||
if entry['value']
|
||||
expression = CountsAxes.time_expression(definition.expression(key),
|
||||
entry['grain'], timezone)
|
||||
scope = scope.where("#{ expression } = ?", entry['value'])
|
||||
end
|
||||
else
|
||||
raise ArgumentError unless definition.axes(history).include?(key)
|
||||
|
||||
scope = scope.where(definition.expression(key) => entry['value'])
|
||||
end
|
||||
scope
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def make_population
|
||||
table = definition.table
|
||||
@scope = @scope.from("#{ table } v").select('v.*')
|
||||
previous = ['rows', 'cols'].any? do |key|
|
||||
QueryState.axes(params.fetch(key, '')).any? { _1['field'] == 'changed_field' }
|
||||
end || (QueryState.records(params, 'drill') + QueryState.records(params, 'conditions'))
|
||||
.any? { _1['field'] == 'changed_field' }
|
||||
if previous
|
||||
keys = definition.keys.map { "previous.#{ _1 } = v.#{ _1 }" }.join(' AND ')
|
||||
@scope = @scope.joins("LEFT JOIN #{ table } previous ON #{ keys } " +
|
||||
'AND previous.version_no = v.version_no - 1 AND ' +
|
||||
access_sql(definition.model, 'previous'))
|
||||
end
|
||||
@scope = @scope.where("v.#{ definition.time_column } <= ?", at)
|
||||
unless history
|
||||
point = timestamp(params['at']) || at
|
||||
raise ArgumentError if point > at
|
||||
|
||||
@scope = @scope.where("v.#{ definition.time_column } <= ?", point)
|
||||
.where(Arel.sql(latest_sql('v', point))).where.not('v.event_type' => 'discard')
|
||||
return
|
||||
end
|
||||
first = timestamp(params['from'])
|
||||
last = timestamp(params['through'], end_date: true)
|
||||
raise ArgumentError if first && last && first > last
|
||||
|
||||
@scope = @scope.where('v.created_at >= ?', first) if first
|
||||
if last
|
||||
operator = params['through'].length == 10 ? '<' : '<='
|
||||
@scope = @scope.where("v.created_at #{ operator } ?", last)
|
||||
end
|
||||
end
|
||||
|
||||
def apply_filters
|
||||
['resource_id', 'user'].each do |key|
|
||||
value = params[key == 'user' ? 'user_id' : key]
|
||||
next if value.nil? || value == ''
|
||||
raise ArgumentError unless value.match?(/\A[0-9]+\z/)
|
||||
|
||||
@scope = @scope.where("#{ definition.expression(key) } = ?", value.to_i)
|
||||
end
|
||||
apply_tags if params['tags'].present? || params['tag'].present?
|
||||
conditions = QueryState.records(params, 'conditions')
|
||||
drills = QueryState.records(params, 'drill')
|
||||
raise ArgumentError unless drills.length <= 100
|
||||
drills.each do |entry|
|
||||
raise ArgumentError unless (entry.keys - ['field', 'grain', 'value']).empty?
|
||||
|
||||
condition = entry.merge('operator' => 'axis_eq')
|
||||
conditions << condition unless conditions.include?(condition)
|
||||
end
|
||||
raise ArgumentError unless conditions.is_a?(Array) && conditions.length <= 100
|
||||
|
||||
values = QueryState.records(params, 'condition_values')
|
||||
values.each do |entry|
|
||||
index = entry.fetch('condition')
|
||||
raise ArgumentError unless index.between?(0, conditions.length - 1)
|
||||
raise ArgumentError unless conditions[index]['values'] == true
|
||||
end
|
||||
evaluated = []
|
||||
conditions.each_with_index do |entry, index|
|
||||
if entry['operator'] == 'axis_eq'
|
||||
next if evaluated.include?(entry)
|
||||
|
||||
evaluated << entry
|
||||
end
|
||||
if entry.key?('values')
|
||||
raise ArgumentError unless entry['values'] == true
|
||||
|
||||
entry = entry.merge('values' => values.select { _1['condition'] == index }
|
||||
.map { _1.fetch('value') })
|
||||
end
|
||||
apply_condition(entry)
|
||||
end
|
||||
|
||||
end
|
||||
|
||||
def apply_condition(entry)
|
||||
@condition ||= CountsCondition.new(self)
|
||||
@scope = @condition.apply(@scope, entry)
|
||||
end
|
||||
|
||||
def snapshot_membership(value)
|
||||
kind, identifier = value.to_s.split(':', 2)
|
||||
raise ArgumentError unless ['internal', 'external'].include?(kind)
|
||||
raise ArgumentError unless identifier&.match?(/\A[1-9]\d*\z/)
|
||||
|
||||
key = kind == 'internal' ? 'tag_id' : 'external_tag_id'
|
||||
definition.model.sanitize_sql_array([
|
||||
"JSON_CONTAINS(v.tags_json, JSON_OBJECT('#{ key }', CAST(? AS UNSIGNED)))",
|
||||
identifier.to_i])
|
||||
end
|
||||
|
||||
def apply_tags
|
||||
raise ArgumentError unless definition.resource == 'posts'
|
||||
|
||||
if params['tag'].present?
|
||||
@scope = @scope.where(snapshot_membership(
|
||||
"#{ params.fetch('kind', 'internal') }:#{ params['tag'] }"))
|
||||
end
|
||||
return unless params['tags'].present?
|
||||
|
||||
raw = params['tags']
|
||||
tokens = raw.split.reject { _1.downcase == 'type:or' }
|
||||
refs = PostTagSearch.parse_references(params, tokens, partial: true)
|
||||
locale = Locale.find_by(code: params.fetch('locale', 'ja'))
|
||||
raise ArgumentError unless locale
|
||||
|
||||
literals = PostTagSearch.literals(locale, tokens.join(' '), references: refs)
|
||||
predicates = literals.map do |literal|
|
||||
reference = literal[:reference]
|
||||
expression = if reference
|
||||
parts = { 'internal' => reference['tag_id'],
|
||||
'external' => reference['external_tag_id'] }.filter_map do |kind, id|
|
||||
snapshot_membership("#{ kind }:#{ id }") if id
|
||||
end
|
||||
parts.empty? ? 'FALSE' : "(#{ parts.join(' OR ') })"
|
||||
else
|
||||
@condition ||= CountsCondition.new(self)
|
||||
@condition.tag_name_match(literal[:name])
|
||||
end
|
||||
literal[:negative] ? "NOT #{ expression }" : expression
|
||||
end
|
||||
return if predicates.empty?
|
||||
|
||||
any = params['match'] == 'any' || raw.split.any? { _1.downcase == 'type:or' }
|
||||
@scope = @scope.where(Arel.sql("(#{ predicates.join(any ? ' OR ' : ' AND ') })"))
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,201 @@
|
||||
# Read-only history queries. SQL identifiers and expressions are server-owned.
|
||||
class CountsQuery
|
||||
AXES = ['month', 'user'].freeze
|
||||
COMMON_COLUMNS = ['resource_id', 'version_no', 'latest', 'event_type', 'user', 'created_at'].freeze
|
||||
COLUMNS = {
|
||||
'posts' => COMMON_COLUMNS + [
|
||||
'thumbnail', 'thumbnail_base', 'title', 'url', 'tags', 'parents', 'video_ms',
|
||||
'original_created_from', 'original_created_before'],
|
||||
'tags' => COMMON_COLUMNS + [
|
||||
'name', 'names', 'category', 'aliases', 'parents', 'deprecated'] }.freeze
|
||||
KEYS = [
|
||||
'locale', 'mode', 'axis', 'page', 'limit', 'from', 'before', 'latest',
|
||||
'user_id', 'resource_id', 'tag', 'columns', 'sort', 'at'].freeze
|
||||
|
||||
attr_reader :page, :limit, :mode, :axis
|
||||
|
||||
def initialize(scope, resource, params, latest_expression: nil)
|
||||
params = CountsResourceDefinition.normalise_params(params)
|
||||
@table = scope.klass.table_name
|
||||
@resource = resource
|
||||
@id_column = resource == 'posts' ? 'post_id' : 'tag_id'
|
||||
@params = params
|
||||
@latest_expression = latest_expression
|
||||
scope = HistoryAccess.visible(scope)
|
||||
@scope = scope.except(:select, :order).joins(
|
||||
"LEFT JOIN users counts_actor ON counts_actor.id = #{ @table }.created_by_user_id")
|
||||
raise ArgumentError unless (params.keys - KEYS).empty?
|
||||
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
|
||||
|
||||
@mode = params.fetch('mode', 'summary')
|
||||
@axis = params.fetch('axis', 'month')
|
||||
raise ArgumentError unless ['summary', 'details'].include?(mode) && AXES.include?(axis)
|
||||
|
||||
@page = integer('page', 1)
|
||||
@limit = integer('limit', 20)
|
||||
raise ArgumentError if limit > 100 || page > 1_000_000
|
||||
|
||||
columns = QueryState.list(params.fetch('columns', COLUMNS.fetch(resource).join(' ')))
|
||||
if columns.empty? || columns.uniq != columns || (columns - COLUMNS.fetch(resource)).any?
|
||||
raise ArgumentError
|
||||
end
|
||||
@sort = parse_sort
|
||||
apply_filters
|
||||
end
|
||||
|
||||
def relation = @scope
|
||||
|
||||
def count = @scope.count
|
||||
|
||||
def latest_expression
|
||||
return @latest_expression if @latest_expression
|
||||
|
||||
at = timestamp('at')
|
||||
cutoff = at ? "AND newer.created_at <= #{ @scope.klass.connection.quote(at) }" : ''
|
||||
<<~SQL.squish
|
||||
NOT EXISTS (SELECT 1 FROM #{ @table } newer
|
||||
WHERE newer.#{ @id_column } = #{ @table }.#{ @id_column }
|
||||
AND newer.version_no > #{ @table }.version_no #{ cutoff })
|
||||
SQL
|
||||
end
|
||||
|
||||
def details
|
||||
@scope.select("#{ @table }.*", Arel.sql("(#{ latest_expression }) AS counts_latest"))
|
||||
.order(Arel.sql(order_sql(detail_expressions, ["#{ @table }.id DESC"])))
|
||||
.limit(limit).offset((page - 1) * limit)
|
||||
end
|
||||
|
||||
def summary(include_count: true)
|
||||
expression = axis == 'month' ? "DATE_FORMAT(#{ @table }.created_at, '%Y-%m')" :
|
||||
"#{ @table }.created_by_user_id"
|
||||
group_order = axis == 'user' ? 'MAX(counts_actor.name)' : expression
|
||||
sql = @scope.group(Arel.sql(expression)).select(Arel.sql(
|
||||
"#{ expression } AS group_key, COUNT(*) AS record_count, " +
|
||||
"#{ group_order } AS group_name")).to_sql
|
||||
connection = @scope.klass.connection
|
||||
source = "FROM (#{ sql }) counts_groups"
|
||||
groups = include_count ? connection.select_value("SELECT COUNT(*) #{ source }") : nil
|
||||
expressions = { 'group' => ['group_name'], 'count' => ['record_count'] }
|
||||
order = order_sql(expressions, ['group_key IS NULL ASC', 'group_key ASC'])
|
||||
rows = connection.select_all(
|
||||
"SELECT group_key, group_name, record_count #{ source } " +
|
||||
"ORDER BY #{ order } LIMIT #{ limit } OFFSET #{ (page - 1) * limit }")
|
||||
{ group_count: groups&.to_i,
|
||||
groups: rows.map do |row|
|
||||
{ key: row['group_key'],
|
||||
name: axis == 'user' ? row['group_name'] : nil,
|
||||
count: row['record_count'].to_i }
|
||||
end }
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def parse_sort
|
||||
value = @params.fetch('sort', mode == 'details' ? 'created_at:desc' : 'group:asc')
|
||||
return [] if value.empty?
|
||||
|
||||
values = QueryState.list(value).map { _1.split(':', -1) }
|
||||
allowed = mode == 'details' ? COLUMNS.fetch(@resource) - ['thumbnail'] : ['group', 'count']
|
||||
invalid = values.any? do |key, direction, *extra|
|
||||
extra.any? || !allowed.include?(key) || !['asc', 'desc'].include?(direction)
|
||||
end
|
||||
if (values.empty? || values.length > allowed.length || invalid ||
|
||||
values.map(&:first).uniq.length != values.length)
|
||||
raise ArgumentError
|
||||
end
|
||||
values
|
||||
end
|
||||
|
||||
# NULL always sorts last, including descending keys. The final key is unique.
|
||||
def order_sql(expressions, final)
|
||||
(@sort.flat_map do |key, direction|
|
||||
expressions.fetch(key).flat_map do |expression|
|
||||
["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"]
|
||||
end
|
||||
end + final).join(', ')
|
||||
end
|
||||
|
||||
def detail_expressions
|
||||
common = { 'resource_id' => ["#{ @table }.#{ @id_column }"],
|
||||
'version_no' => ["#{ @table }.version_no"],
|
||||
'latest' => ["(#{ latest_expression })"],
|
||||
'user' => ['counts_actor.name', "#{ @table }.created_by_user_id"],
|
||||
'created_at' => ["#{ @table }.created_at"] }
|
||||
if @resource == 'posts'
|
||||
fields = [
|
||||
'title', 'url', 'thumbnail_base', 'video_ms',
|
||||
'original_created_from', 'original_created_before']
|
||||
common.merge(fields.to_h { [_1, ["#{ @table }.#{ _1 }"]] }).merge(
|
||||
'tags' => json_order('post_versions.tags_json'),
|
||||
'parents' => list_order('post_versions.parent_post_ids'))
|
||||
else
|
||||
common.merge(
|
||||
'name' => ['tag_versions.name'],
|
||||
'names' => json_order('tag_versions.names'),
|
||||
'category' => ['tag_versions.category'],
|
||||
'aliases' => json_order(
|
||||
"JSON_EXTRACT(tag_versions.names, '$.*.aliases[*]')"),
|
||||
'parents' => list_order('tag_versions.parent_tag_ids'),
|
||||
'deprecated' => ['tag_versions.deprecated_at'])
|
||||
end
|
||||
end
|
||||
|
||||
# Multi-valued parent cells sort by cardinality, then binary snapshot text.
|
||||
def json_order(column)
|
||||
[
|
||||
"COALESCE(JSON_LENGTH(#{ column }), 0)",
|
||||
"COALESCE(CAST(#{ column } AS CHAR), '[]') COLLATE utf8mb4_bin"]
|
||||
end
|
||||
|
||||
def list_order(column)
|
||||
[
|
||||
"CASE WHEN #{ column } = '' THEN 0 ELSE " +
|
||||
"1 + LENGTH(#{ column }) - LENGTH(REPLACE(#{ column }, ' ', '')) END",
|
||||
"#{ column } COLLATE utf8mb4_bin"]
|
||||
end
|
||||
|
||||
def integer(key, default = nil)
|
||||
value = @params[key].presence
|
||||
return default if value.nil? && !default.nil?
|
||||
raise ArgumentError unless value&.match?(/\A[1-9][0-9]{0,9}\z/)
|
||||
|
||||
value.to_i
|
||||
end
|
||||
|
||||
def timestamp(key)
|
||||
value = @params[key].presence
|
||||
return nil unless value
|
||||
pattern = /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d{1,6})?)?Z\z/
|
||||
raise ArgumentError unless value.match?(pattern)
|
||||
|
||||
Date.iso8601(value.first(10))
|
||||
Time.iso8601(value.length == 17 ? value.sub(/Z\z/, ':00Z') : value)
|
||||
end
|
||||
|
||||
def apply_filters
|
||||
at = timestamp('at')
|
||||
first = timestamp('from')
|
||||
last = timestamp('before')
|
||||
raise ArgumentError if first && last && first >= last
|
||||
raise ArgumentError unless ['0', '1'].include?(@params.fetch('latest', '1'))
|
||||
|
||||
@scope = @scope.where("#{ @table }.created_at <= ?", at) if at
|
||||
@scope = @scope.where(Arel.sql(latest_expression)) unless @params['latest'] == '0'
|
||||
@scope = @scope.where("#{ @table }.created_at >= ?", first) if first
|
||||
@scope = @scope.where("#{ @table }.created_at < ?", last) if last
|
||||
if @params['resource_id'].present?
|
||||
@scope = @scope.where(@table => { @id_column => integer('resource_id') })
|
||||
end
|
||||
if @params['user_id'].present?
|
||||
user = @params['user_id'] == 'none' ? nil : integer('user_id')
|
||||
@scope = @scope.where(@table => { created_by_user_id: user })
|
||||
end
|
||||
return unless @params['tag'].present?
|
||||
raise ArgumentError unless @resource == 'posts'
|
||||
|
||||
id = integer('tag')
|
||||
@scope = @scope.where(
|
||||
"JSON_CONTAINS(post_versions.tags_json, JSON_OBJECT('tag_id', CAST(? AS UNSIGNED))) OR " +
|
||||
"JSON_CONTAINS(prev.tags_json, JSON_OBJECT('tag_id', CAST(? AS UNSIGNED)))", id, id)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,309 @@
|
||||
# Server-owned version identities, attributes and field types. No client SQL is accepted.
|
||||
class CountsResourceDefinition
|
||||
# Compatibility belongs at the request boundary, not in field definitions.
|
||||
def self.normalise_params(params)
|
||||
aliases = nil
|
||||
predicates = nil
|
||||
scope = params['name_scope']
|
||||
raise ArgumentError if scope && !['locale', 'language'].include?(scope)
|
||||
params.each_with_object({}) do |(key, value), result|
|
||||
result[key] = value
|
||||
next unless value.is_a?(String)
|
||||
|
||||
if ['columns', 'order', 'sort', 'rows', 'cols', 'axis'].include?(key)
|
||||
result[key] = value.split(/ /, -1).map { |token|
|
||||
token.sub(/\Ais_latest(?=:|\z)/, 'latest')
|
||||
}.join(' ')
|
||||
elsif (key.match?(/\A(?:conditions|drill)\.[0-9]+\.field\z/) &&
|
||||
value == 'is_latest')
|
||||
result[key] = 'latest'
|
||||
end
|
||||
if (['columns', 'children', 'order', 'sort', 'child_sort'].include?(key) &&
|
||||
value.match?(/(?:name|aliases)_[a-z]+_[a-z0-9_]+/))
|
||||
aliases ||= new('tags').column_aliases.to_h { [_1[:key], _1[:value]] }
|
||||
result[key] = result[key].gsub(/(?:name|aliases)_[a-z0-9_]+/) { aliases.fetch(_1, _1) }
|
||||
end
|
||||
if (scope != 'language' && key.match?(/\Aconditions\.[0-9]+\.field\z/) &&
|
||||
value.match?(/(?:name|aliases)_[a-z0-9_]+/))
|
||||
predicates ||= new('tags').condition_aliases.to_h { [_1[:key], _1[:value]] }
|
||||
result[key] = value.gsub(/(?:name|aliases)_[a-z0-9_]+/) { predicates.fetch(_1, _1) }
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
COMMON = { 'resource_id' => :integer, 'version_no' => :integer,
|
||||
'event_type' => :enum, 'user' => :integer, 'created_at' => :datetime }.freeze
|
||||
DEFINITIONS = {
|
||||
'posts' => [PostVersion, ['post_id'], {
|
||||
'title' => :string, 'url' => :string, 'thumbnail_base' => :string,
|
||||
'tags_json' => :json, 'parent_post_ids' => :string, 'video_ms' => :integer,
|
||||
'original_created_from' => :datetime, 'original_created_before' => :datetime }],
|
||||
'tags' => [TagVersion, ['tag_id'], {
|
||||
'name' => :string, 'names' => :json, 'category' => :enum, 'aliases' => :string,
|
||||
'parent_tag_ids' => :string, 'deprecated_at' => :datetime }],
|
||||
'nico_tags' => [NicoTagVersion, ['tag_id'], {
|
||||
'name' => :string, 'linked_tags' => :string }],
|
||||
'tag_descriptions' => [TagDescriptionVersion, ['tag_id', 'language_code'], {
|
||||
'language_code' => :enum, 'body' => :string,
|
||||
'source_language_code' => :enum, 'source_version_no' => :integer }],
|
||||
'tag_groups' => [TagGroupVersion, ['tag_group_id'], {
|
||||
'name' => :string, 'description' => :string,
|
||||
'members' => :json, 'placements' => :json }],
|
||||
'materials' => [MaterialVersion, ['material_id'], {
|
||||
'url' => :string, 'parent_id' => :integer, 'tag_id' => :integer,
|
||||
'tag_name' => :string, 'tag_category' => :enum,
|
||||
'discarded_at' => :datetime, 'updated_at' => :datetime,
|
||||
'updated_by_user_id' => :integer, 'export_paths_json' => :json,
|
||||
'file_blob_id' => :integer, 'file_filename' => :string,
|
||||
'file_content_type' => :string, 'file_byte_size' => :integer,
|
||||
'file_checksum' => :string, 'file_sha256' => :string,
|
||||
'source_kind' => :enum, 'source_uri' => :string, 'source_path' => :string,
|
||||
'source_file_id' => :string, 'normalized_source_key' => :string }] }.freeze
|
||||
OPERATORS = {
|
||||
string: ['eq', 'ne', 'like', 'in', 'not_in'],
|
||||
enum: ['eq', 'ne', 'in', 'not_in'], boolean: ['eq', 'ne', 'in', 'not_in'],
|
||||
integer: ['eq', 'ne', 'gt', 'lt', 'ge', 'le', 'range', 'in', 'not_in'],
|
||||
datetime: ['eq', 'ne', 'gt', 'lt', 'ge', 'le', 'range', 'in', 'not_in'],
|
||||
interval: ['range'] }.freeze
|
||||
LABELS = {
|
||||
'resource_id' => 'counts:columns.resourceId', 'version_no' => 'counts:columns.version',
|
||||
'latest' => 'counts:columns.latest', 'event_type' => 'counts:eventType',
|
||||
'user' => 'history:actor', 'created_at' => 'counts:columns.changedAt',
|
||||
'thumbnail' => 'counts:columns.thumbnailDisplay',
|
||||
'thumbnail_base' => 'counts:columns.thumbnailURL',
|
||||
'title' => 'counts:columns.postTitle', 'url' => 'counts:columns.postURL',
|
||||
'tags' => 'counts:columns.tags', 'tags_json' => 'counts:columns.tags',
|
||||
'video_ms' => 'shared:labels.duration', 'name' => 'counts:columns.name',
|
||||
'names' => 'counts:columns.names', 'category' => 'common:categoryLabel',
|
||||
'aliases' => 'tags:aliases', 'deprecated' => 'counts:columns.deprecated',
|
||||
'original_created_from' => 'counts:columns.originalFrom',
|
||||
'original_created_before' => 'counts:columns.originalBefore',
|
||||
'language_code' => 'counts:columns.language' }.freeze
|
||||
CHILD_LABELS = {
|
||||
'id' => 'counts:columns.resourceId', 'versionNo' => 'history:version',
|
||||
'kind' => 'counts:columns.kind', 'name' => 'history:name',
|
||||
'category' => 'common:categoryLabel', 'sections' => 'counts:columns.sections',
|
||||
'names' => 'counts:columns.names', 'aliases' => 'tags:aliases',
|
||||
'languageCode' => 'counts:columns.language',
|
||||
'scriptCode' => 'history:nameChanges.script',
|
||||
'autoGenerated' => 'history:nameChanges.automatic',
|
||||
'tagId' => 'counts:attributes.tag_id', 'tagGroupId' => 'counts:resources.tag_groups',
|
||||
'sectionName' => 'counts:attributes.section_name',
|
||||
'description' => 'counts:attributes.description',
|
||||
'children' => 'counts:attributes.children', 'profile' => 'counts:attributes.profile',
|
||||
'path' => 'counts:attributes.path' }.freeze
|
||||
|
||||
attr_reader :model, :keys, :fields, :resource
|
||||
|
||||
def initialize(resource)
|
||||
@resource = resource
|
||||
@model, @keys, attributes = DEFINITIONS.fetch(resource) { raise ArgumentError }
|
||||
@fields = COMMON.merge(attributes)
|
||||
end
|
||||
|
||||
def table = model.table_name
|
||||
def time_column = 'created_at'
|
||||
def column(key)
|
||||
raise ArgumentError unless fields.key?(key)
|
||||
|
||||
{ 'resource_id' => keys.first, 'user' => 'created_by_user_id' }.fetch(key, key)
|
||||
end
|
||||
|
||||
def expression(key, row = 'v') = "#{ row }.#{ column(key) }"
|
||||
def changes = fields.keys - COMMON.keys - ['updated_at', 'updated_by_user_id']
|
||||
def axes(history)
|
||||
fields.reject { |_key, type| type == :json }.keys +
|
||||
(resource == 'posts' ? ['tags'] : []) + (history ? ['changed_field'] : [])
|
||||
end
|
||||
|
||||
def metadata
|
||||
fields.map do |key, type|
|
||||
{ key:, type:, nullable: model.columns_hash.fetch(column(key)).null,
|
||||
operators: OPERATORS.fetch(type, []) }
|
||||
end
|
||||
end
|
||||
|
||||
def field_label(key) = LABELS.fetch(key, "counts:attributes.#{ key }")
|
||||
|
||||
def self.name_key(code) = "name_#{ code.to_s.downcase.tr('-', '_') }"
|
||||
def self.aliases_key(code) = "aliases_#{ code.to_s.downcase.tr('-', '_') }"
|
||||
|
||||
def name_columns
|
||||
@name_columns ||= begin
|
||||
codes = Language.select(:code).order(:created_at, :code).map do |language|
|
||||
{ code: language.code, language_code: language.code }
|
||||
end
|
||||
names = codes.map do |locale|
|
||||
{ key: self.class.name_key(locale[:code]), label: 'counts:localeName',
|
||||
name_locale: locale }
|
||||
end
|
||||
aliases = codes.map do |locale|
|
||||
{ key: self.class.aliases_key(locale[:code]), label: 'counts:localeAliases',
|
||||
name_locale: locale, multiple_values: true }
|
||||
end
|
||||
names + aliases
|
||||
end
|
||||
end
|
||||
|
||||
# Old locale-qualified predicates keep their script restriction at the input boundary.
|
||||
def legacy_name_columns
|
||||
@legacy_name_columns ||= Locale.select(:code, :language_code, :script_code)
|
||||
.order(:code).flat_map do |locale|
|
||||
codes = { code: locale.code, language_code: locale.language_code,
|
||||
script_code: locale.script_code }
|
||||
code = locale.code == locale.language_code ? "#{ locale.code }_#{ locale.script_code }" :
|
||||
locale.code
|
||||
[{ key: self.class.name_key(code), label: 'counts:localeName',
|
||||
name_locale: codes, legacy: true },
|
||||
{ key: self.class.aliases_key(code), label: 'counts:localeAliases',
|
||||
name_locale: codes, multiple_values: true, legacy: true }]
|
||||
end
|
||||
end
|
||||
|
||||
def column_aliases
|
||||
legacy_name_columns.map do |field|
|
||||
language = field[:name_locale][:language_code]
|
||||
value = if field[:multiple_values]
|
||||
self.class.aliases_key(language)
|
||||
else self.class.name_key(language)
|
||||
end
|
||||
{ key: field[:key], value: }
|
||||
end
|
||||
end
|
||||
|
||||
def condition_aliases
|
||||
legacy_name_columns.filter_map do |field|
|
||||
code = field[:name_locale][:code]
|
||||
key = field[:multiple_values] ? self.class.aliases_key(code) : self.class.name_key(code)
|
||||
next if key == field[:key]
|
||||
|
||||
{ key:, value: field[:key] }
|
||||
end
|
||||
end
|
||||
|
||||
def name_column(key)
|
||||
name_columns.find { _1[:key] == key } || legacy_name_columns.find { _1[:key] == key }
|
||||
end
|
||||
|
||||
def name_expression(key, row = 'v')
|
||||
if key.start_with?('aliases_')
|
||||
source, value = alias_source(key, row)
|
||||
return "CAST(COALESCE((SELECT JSON_ARRAYAGG(#{ value }) OVER (" +
|
||||
'ORDER BY locale_alias.ordinal ROWS BETWEEN UNBOUNDED PRECEDING ' +
|
||||
"AND UNBOUNDED FOLLOWING) FROM #{ source } WHERE #{ value } IS NOT NULL " +
|
||||
'LIMIT 1), JSON_ARRAY()) AS CHAR) COLLATE utf8mb4_bin'
|
||||
end
|
||||
locale = name_column(key).fetch(:name_locale)
|
||||
connection = model.connection
|
||||
path = connection.quote("$.#{ locale[:language_code].to_json }")
|
||||
value = "JSON_UNQUOTE(NULLIF(JSON_EXTRACT(#{ row }.names, " +
|
||||
"CONCAT(#{ path }, '.name')), CAST('null' AS JSON)))"
|
||||
return value unless locale[:script_code]
|
||||
|
||||
script = connection.quote(locale[:script_code])
|
||||
"CASE WHEN LOWER(JSON_UNQUOTE(JSON_EXTRACT(#{ row }.names, " +
|
||||
"CONCAT(#{ path }, '.script_code')))) = LOWER(#{ script }) " +
|
||||
"THEN #{ value } END"
|
||||
end
|
||||
|
||||
def alias_source(key, row = 'v')
|
||||
locale = name_column(key).fetch(:name_locale)
|
||||
connection = model.connection
|
||||
path = connection.quote("$.#{ locale[:language_code].to_json }.aliases[*]")
|
||||
source = "JSON_TABLE(COALESCE(#{ row }.names, JSON_OBJECT()), #{ path } " +
|
||||
"COLUMNS (ordinal FOR ORDINALITY, name TEXT PATH '$.name', " +
|
||||
"script TEXT PATH '$.script_code')) locale_alias"
|
||||
return [source, 'locale_alias.name'] unless locale[:script_code]
|
||||
|
||||
script = connection.quote(locale[:script_code])
|
||||
value = "CASE WHEN LOWER(locale_alias.script) = LOWER(#{ script }) " +
|
||||
'THEN locale_alias.name END'
|
||||
[source, value]
|
||||
end
|
||||
|
||||
def output_fields
|
||||
content = case resource
|
||||
when 'posts'
|
||||
['thumbnail', 'title', 'url', 'thumbnail_base', 'tags', 'parents', 'video_ms',
|
||||
'original_created_from', 'original_created_before']
|
||||
when 'tags' then ['name', 'names', 'category', 'aliases', 'parents', 'deprecated']
|
||||
else fields.keys - COMMON.keys
|
||||
end
|
||||
keys = ['resource_id', 'version_no', 'latest'] + (content - ['thumbnail']) +
|
||||
['event_type', 'user', 'created_at']
|
||||
keys.unshift('thumbnail') if content.include?('thumbnail')
|
||||
keys.flat_map do |key|
|
||||
next name_columns if key == 'name' && resource == 'tags'
|
||||
next [] if key == 'names' && resource == 'tags'
|
||||
next [] if key == 'aliases' && resource == 'tags'
|
||||
label = if key == 'parents'
|
||||
resource == 'tags' ? 'tags:parents' : 'shared:labels.parentPosts'
|
||||
else field_label(key)
|
||||
end
|
||||
field = { key:, label: }
|
||||
children = child_fields(key)
|
||||
field[:children] = children if children.any?
|
||||
[field]
|
||||
end
|
||||
end
|
||||
|
||||
def child_fields(key, depth = 4)
|
||||
keys = case key
|
||||
when 'tags' then ['id', 'versionNo', 'kind', 'name', 'category', 'sections', 'aliases']
|
||||
when 'names' then ['languageCode', 'scriptCode', 'name', 'aliases', 'autoGenerated']
|
||||
when 'aliases' then ['languageCode', 'name', 'scriptCode']
|
||||
when 'parents' then resource == 'tags' ? ['id', 'name'] : ['id']
|
||||
when 'members' then ['tagId', 'tagGroupId', 'versionNo']
|
||||
when 'export_paths_json' then ['profile', 'path']
|
||||
when 'placements', 'children'
|
||||
scalar = ['sectionName', 'description', 'tagId', 'tagGroupId']
|
||||
scalar + (depth > 0 ? ['children'] : [])
|
||||
else []
|
||||
end
|
||||
keys.flat_map do |child|
|
||||
next [] if child == 'aliases' && key == 'tags'
|
||||
if child == 'name' && ['tags', 'parents'].include?(key)
|
||||
next name_columns
|
||||
end
|
||||
label = key == 'members' && child == 'tagId' ?
|
||||
'counts:columns.resourceId' : CHILD_LABELS.fetch(child)
|
||||
field = { key: child, label: }
|
||||
children = child_fields(child, child == 'children' ? depth - 1 : depth)
|
||||
# Language is attached to top-level aliases, not to aliases inside a language entry.
|
||||
if child == 'aliases' && key == 'names'
|
||||
children = children.reject { _1[:key] == 'languageCode' }
|
||||
end
|
||||
field[:children] = children if children.any?
|
||||
[field]
|
||||
end
|
||||
end
|
||||
|
||||
def condition_fields
|
||||
fields = output_fields
|
||||
if resource == 'tags'
|
||||
fields = fields.map do |field|
|
||||
next field unless field[:key] == 'parents'
|
||||
|
||||
field.merge(children: field[:children] + legacy_name_columns + [
|
||||
{ key: 'name', label: 'history:name' },
|
||||
{ key: 'names', label: 'counts:columns.names', children: child_fields('names') }])
|
||||
end
|
||||
fields += legacy_name_columns + [{ key: 'name', label: 'counts:columns.name' },
|
||||
{ key: 'names', label: 'counts:columns.names', children: child_fields('names') },
|
||||
{ key: 'aliases', label: 'tags:aliases', children: child_fields('aliases') }]
|
||||
end
|
||||
if resource == 'posts'
|
||||
fields = fields.map do |field|
|
||||
next field unless field[:key] == 'tags'
|
||||
|
||||
field.merge(children: field[:children] + legacy_name_columns + [
|
||||
{ key: 'name', label: 'history:name' },
|
||||
{ key: 'names', label: 'counts:columns.names', children: child_fields('names') },
|
||||
{ key: 'aliases', label: 'tags:aliases', children: child_fields('aliases') }])
|
||||
end
|
||||
end
|
||||
fields
|
||||
end
|
||||
|
||||
def filter_metadata = @filter_metadata ||= CountsCondition.fields(self)
|
||||
end
|
||||
@@ -0,0 +1,370 @@
|
||||
# Restores one recorded version per resource before applying record filters.
|
||||
class CountsStateQuery
|
||||
KEYS = ['locale', 'resource', 'mode', 'at', 'from', 'before', 'grain',
|
||||
'kind', 'tag', 'page', 'limit', 'sort', 'axis', 'user_id', 'resource_id'].freeze
|
||||
MAX_PERIODS = 366
|
||||
|
||||
attr_reader :at
|
||||
|
||||
def initialize(params, language_code:)
|
||||
raise ArgumentError unless (params.keys - KEYS).empty?
|
||||
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
|
||||
|
||||
@params = params
|
||||
@connection = ApplicationRecord.connection
|
||||
@language = language_code
|
||||
@resource = params.fetch('resource', 'posts')
|
||||
@mode = params.fetch('mode', 'summary')
|
||||
@inclusive = ['summary', 'details'].include?(@mode)
|
||||
@axis = params.fetch('axis', @resource == 'posts' ? 'tags' : 'month')
|
||||
@kind = params.fetch('kind', 'internal')
|
||||
@grain = params.fetch('grain', 'month')
|
||||
raise ArgumentError unless ['posts', 'tags'].include?(@resource)
|
||||
raise ArgumentError unless ['summary', 'details', 'state', 'trend', 'snapshot'].include?(@mode)
|
||||
raise ArgumentError unless ['internal', 'external', 'all'].include?(@kind)
|
||||
raise ArgumentError unless ['day', 'month', 'year'].include?(@grain)
|
||||
|
||||
raise ArgumentError unless ['tags', 'month', 'user'].include?(@axis)
|
||||
if @mode == 'summary' && @axis == 'tags' && @resource != 'posts'
|
||||
raise ArgumentError
|
||||
end
|
||||
|
||||
@page = integer('page', 1)
|
||||
@limit = integer('limit', 20)
|
||||
@tag = integer('tag') if params['tag'].present?
|
||||
raise ArgumentError if @limit > 100 || @page > 1_000_000
|
||||
raise ArgumentError if @tag && @kind == 'all'
|
||||
raise ArgumentError if @mode == 'trend' && !@tag
|
||||
|
||||
@now = Time.current.utc
|
||||
@at = timestamp('at') || @now
|
||||
raise ArgumentError if @at > @now
|
||||
end
|
||||
|
||||
def call(metadata: true)
|
||||
@metadata = metadata
|
||||
# Keep totals, rows, and detail pages on one repeatable-read database snapshot.
|
||||
ApplicationRecord.transaction do
|
||||
unless @history_loaded
|
||||
@post_start = HistoryAccess.visible(PostVersion.all).minimum(:created_at)
|
||||
@tag_start = HistoryAccess.visible(TagVersion.all).minimum(:created_at)
|
||||
@history_loaded = true
|
||||
end
|
||||
result = { population_count: HistoryAccess.population_count(
|
||||
@resource == 'posts' ? PostVersion : TagVersion), page: @page, limit: @limit, at: @at.iso8601(6),
|
||||
coverage: { basis: 'recorded_snapshots', exact: false,
|
||||
post_history_from: @post_start&.iso8601(6),
|
||||
tag_history_from: @tag_start&.iso8601(6) } }
|
||||
case @mode
|
||||
when 'trend'
|
||||
result.merge(trend)
|
||||
when 'summary'
|
||||
result.merge(metadata ? { totals: totals } : {}).merge(summary)
|
||||
when 'details', 'snapshot'
|
||||
result.merge(metadata ? { totals: totals } : {}).merge(details)
|
||||
else
|
||||
result.merge(metadata ? { totals: totals } : {}).merge(listing)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def each_page
|
||||
return enum_for(:each_page) unless block_given?
|
||||
|
||||
@page = 1
|
||||
first = call
|
||||
yield first
|
||||
total = ['summary', 'state', 'trend'].include?(@mode) ? first[:group_count] : first[:count]
|
||||
while @page * @limit < total
|
||||
@page += 1
|
||||
yield call(metadata: false)
|
||||
end
|
||||
end
|
||||
|
||||
def validate_query!
|
||||
filtered_scope(@resource)
|
||||
case @mode
|
||||
when 'summary'
|
||||
if @axis == 'tags'
|
||||
order_sql({ 'name' => 'name', 'count' => 'post_count', 'tag_id' => 'tag_id',
|
||||
'kind' => 'kind', 'deprecated' => 'deprecated_at' }, 'count:desc', [])
|
||||
else
|
||||
record_query('summary')
|
||||
end
|
||||
when 'details', 'snapshot'
|
||||
record_query('details')
|
||||
end
|
||||
self
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def quote(value) = @connection.quote(value)
|
||||
|
||||
def integer(key, default = nil)
|
||||
value = @params[key]
|
||||
return default if value.blank? && default
|
||||
raise ArgumentError unless value&.match?(/\A[1-9][0-9]{0,9}\z/)
|
||||
|
||||
value.to_i
|
||||
end
|
||||
|
||||
def timestamp(key)
|
||||
value = @params[key].presence
|
||||
return nil unless value
|
||||
pattern = /\A\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d{1,6})?)?Z\z/
|
||||
raise ArgumentError unless value.match?(pattern)
|
||||
|
||||
Date.iso8601(value.first(10))
|
||||
Time.iso8601(value.length == 17 ? value.sub(/Z\z/, ':00Z') : value).utc
|
||||
end
|
||||
|
||||
# Version numbers order equal-timestamp writes. Do not filter tags before this selection.
|
||||
def latest_condition(table, row, boundary)
|
||||
key = table == 'post_versions' ? 'post_id' : 'tag_id'
|
||||
<<~SQL.squish
|
||||
#{ row }.created_at #{ @inclusive ? '<=' : '<' } #{ boundary } AND NOT EXISTS (
|
||||
SELECT 1 FROM #{ table } newer
|
||||
WHERE newer.#{ key } = #{ row }.#{ key }
|
||||
AND newer.created_at #{ @inclusive ? '<=' : '<' } #{ boundary }
|
||||
AND newer.version_no > #{ row }.version_no)
|
||||
SQL
|
||||
end
|
||||
|
||||
def state_sql(table, boundary, live: false)
|
||||
model = { 'post_versions' => PostVersion, 'tag_versions' => TagVersion,
|
||||
'nico_tag_versions' => NicoTagVersion }.fetch(table)
|
||||
sql = "SELECT v.* FROM #{ table } v WHERE #{ latest_condition(table, 'v', boundary) } " +
|
||||
"AND (#{ HistoryAccess.sql(model, 'v') })"
|
||||
live ? "#{ sql } AND v.event_type <> 'discard'" : sql
|
||||
end
|
||||
|
||||
def membership(row)
|
||||
key = @kind == 'external' ? 'external_tag_id' : 'tag_id'
|
||||
"JSON_CONTAINS(#{ row }.tags_json, JSON_OBJECT('#{ key }', #{ @tag }))"
|
||||
end
|
||||
|
||||
# Deprecation keeps a tag in existence. Only an explicit discard removes it.
|
||||
def tag_visible(boundary)
|
||||
return '1 = 1' if @kind == 'external'
|
||||
|
||||
<<~SQL.squish
|
||||
NOT EXISTS (SELECT 1 FROM tag_versions t
|
||||
WHERE t.tag_id = #{ @tag } AND t.event_type = 'discard'
|
||||
AND #{ latest_condition('tag_versions', 't', boundary) })
|
||||
SQL
|
||||
end
|
||||
|
||||
def ctes
|
||||
boundary = quote(@at)
|
||||
<<~SQL
|
||||
WITH post_at AS (#{ filtered_scope('posts').to_sql }),
|
||||
tag_at AS (#{ state_sql('tag_versions', boundary) }),
|
||||
nico_at AS (#{ state_sql('nico_tag_versions', boundary) }),
|
||||
pairs AS (
|
||||
SELECT DISTINCT p.post_id,
|
||||
CASE WHEN entry.tag_id IS NULL THEN 'external' ELSE 'internal' END AS kind,
|
||||
COALESCE(entry.tag_id, entry.external_tag_id) AS tag_id
|
||||
FROM post_at p
|
||||
JOIN JSON_TABLE(CAST(p.tags_json AS JSON), '$[*]' COLUMNS (
|
||||
tag_id BIGINT PATH '$.tag_id' NULL ON EMPTY ERROR ON ERROR,
|
||||
external_tag_id BIGINT PATH '$.external_tag_id' NULL ON EMPTY ERROR ON ERROR
|
||||
)) entry
|
||||
LEFT JOIN tag_at t ON t.tag_id = entry.tag_id
|
||||
WHERE COALESCE(entry.tag_id, entry.external_tag_id) IS NOT NULL
|
||||
AND (t.event_type IS NULL OR t.event_type <> 'discard')
|
||||
)
|
||||
SQL
|
||||
end
|
||||
|
||||
def totals
|
||||
row = @connection.select_one(<<~SQL)
|
||||
#{ ctes }
|
||||
SELECT (SELECT COUNT(*) FROM post_at) AS posts,
|
||||
(SELECT COUNT(*) FROM (#{ filtered_scope('tags').to_sql }) counted_tags) AS tags,
|
||||
(SELECT COUNT(*) FROM tag_at WHERE event_type <> 'discard'
|
||||
AND deprecated_at IS NOT NULL) AS deprecated_tags,
|
||||
(SELECT COUNT(*) FROM pairs) AS assignments,
|
||||
(SELECT COUNT(*) FROM pairs WHERE kind = 'external') AS external_assignments
|
||||
SQL
|
||||
row.transform_values!(&:to_i)
|
||||
row.merge('posts' => available?(@post_start, @at) ? row['posts'] : nil,
|
||||
'assignments' => available?(@post_start, @at) ? row['assignments'] : nil,
|
||||
'external_assignments' => available?(@post_start, @at) ?
|
||||
row['external_assignments'] : nil,
|
||||
'tags' => available?(@tag_start, @at) ? row['tags'] : nil,
|
||||
'deprecated_tags' => available?(@tag_start, @at) ? row['deprecated_tags'] : nil)
|
||||
end
|
||||
|
||||
def available?(first, boundary)
|
||||
first && (@inclusive ? first <= boundary : first < boundary)
|
||||
end
|
||||
|
||||
def listing
|
||||
catalog = <<~SQL
|
||||
, tag_counts AS (
|
||||
SELECT membership.kind, membership.tag_id, COUNT(*) AS post_count
|
||||
FROM pairs membership GROUP BY membership.kind, membership.tag_id
|
||||
), catalog AS (
|
||||
SELECT 'internal' AS kind, tag_id, name, names,
|
||||
deprecated_at FROM tag_at WHERE event_type <> 'discard'
|
||||
UNION ALL
|
||||
SELECT tc.kind, tc.tag_id, external_version.name, JSON_OBJECT() AS names,
|
||||
NULL AS deprecated_at
|
||||
FROM tag_counts tc
|
||||
LEFT JOIN tag_at internal_version
|
||||
ON tc.kind = 'internal' AND internal_version.tag_id = tc.tag_id
|
||||
LEFT JOIN nico_at external_version
|
||||
ON tc.kind = 'external' AND external_version.tag_id = tc.tag_id
|
||||
WHERE tc.kind = 'external' OR internal_version.tag_id IS NULL
|
||||
)
|
||||
SQL
|
||||
filters = []
|
||||
filters << "c.kind = #{ quote(@kind) }" unless @kind == 'all'
|
||||
filters << "c.tag_id = #{ @tag }" if @tag
|
||||
where = filters.any? ? "WHERE #{ filters.join(' AND ') }" : ''
|
||||
source = 'FROM catalog c LEFT JOIN tag_counts n ON n.kind = c.kind AND n.tag_id = c.tag_id'
|
||||
prefix = "#{ ctes } #{ catalog }"
|
||||
total = if @metadata
|
||||
@connection.select_value("#{ prefix } SELECT COUNT(*) #{ source } #{ where }").to_i
|
||||
end
|
||||
order = order_sql({ 'name' => 'c.name', 'count' => 'COALESCE(n.post_count, 0)',
|
||||
'tag_id' => 'c.tag_id', 'kind' => 'c.kind',
|
||||
'deprecated' => 'c.deprecated_at' }, 'count:desc',
|
||||
['c.kind ASC', 'c.tag_id ASC'])
|
||||
rows = @connection.select_all(<<~SQL)
|
||||
#{ prefix }
|
||||
SELECT c.kind, c.tag_id, c.name, c.names, c.deprecated_at,
|
||||
COALESCE(n.post_count, 0) AS post_count #{ source } #{ where }
|
||||
ORDER BY #{ order } LIMIT #{ @limit } OFFSET #{ (@page - 1) * @limit }
|
||||
SQL
|
||||
{ count: total, group_count: total, state_rows: rows.map do |row|
|
||||
{ kind: row['kind'], tag_id: row['tag_id'], name: row['name'],
|
||||
names: row['names'].is_a?(String) ? JSON.parse(row['names']) : row['names'],
|
||||
deprecated: row['deprecated_at']&.iso8601,
|
||||
count: available?(@post_start, @at) ? row['post_count'].to_i : nil,
|
||||
at: @at.iso8601(6) }
|
||||
end }
|
||||
end
|
||||
|
||||
# All record filters apply after selecting the latest recorded version at the point.
|
||||
def restored_scope(resource)
|
||||
model = resource == 'posts' ? PostVersion : TagVersion
|
||||
table = model.table_name
|
||||
scope = model.where(Arel.sql(latest_condition(table, table, quote(@at))))
|
||||
.where.not(event_type: 'discard')
|
||||
if @tag
|
||||
scope = if resource == 'posts'
|
||||
scope.where("#{ membership(table) } AND #{ tag_visible(quote(@at)) }")
|
||||
elsif @kind == 'internal'
|
||||
scope.where(tag_id: @tag)
|
||||
else
|
||||
scope.none
|
||||
end
|
||||
end
|
||||
scope
|
||||
end
|
||||
|
||||
def filtered_scope(resource)
|
||||
params = @params.slice('from', 'before', 'user_id', 'resource_id')
|
||||
.merge('mode' => 'details', 'latest' => '0', 'sort' => '')
|
||||
CountsQuery.new(restored_scope(resource), resource, params,
|
||||
latest_expression: '1').relation
|
||||
end
|
||||
|
||||
def record_query(mode)
|
||||
params = @params.slice('locale', 'sort', 'page', 'limit')
|
||||
.merge('mode' => mode, 'latest' => '0', 'axis' => @axis == 'tags' ? 'month' : @axis,
|
||||
'page' => @page.to_s, 'limit' => @limit.to_s)
|
||||
CountsQuery.new(filtered_scope(@resource), @resource, params, latest_expression: '1')
|
||||
end
|
||||
|
||||
def summary
|
||||
return listing if @axis == 'tags'
|
||||
|
||||
query = record_query('summary')
|
||||
{ count: @metadata ? query.count : nil }.merge(query.summary(include_count: @metadata))
|
||||
end
|
||||
|
||||
def details
|
||||
query = record_query('details')
|
||||
{ count: @metadata ? query.count : nil,
|
||||
versions: CountsVersionRepr.many(query.details, resource: @resource,
|
||||
language_code: @language, at: @at,
|
||||
inclusive: @inclusive) }
|
||||
end
|
||||
|
||||
def next_boundary(time)
|
||||
case @grain
|
||||
when 'day' then time.beginning_of_day + 1.day
|
||||
when 'month' then time.beginning_of_month.next_month
|
||||
else time.beginning_of_year.next_year
|
||||
end
|
||||
end
|
||||
|
||||
def periods
|
||||
@before = timestamp('before') || @now.beginning_of_month
|
||||
@from = timestamp('from') || @before.prev_year
|
||||
raise ArgumentError if @from >= @before || @before > @now
|
||||
|
||||
rows = []
|
||||
boundary = next_boundary(@from)
|
||||
while boundary <= @before
|
||||
raise ArgumentError if rows.length >= MAX_PERIODS
|
||||
|
||||
format = { 'day' => '%Y-%m-%d', 'month' => '%Y-%m', 'year' => '%Y' }.fetch(@grain)
|
||||
rows << [boundary, (boundary - 1.second).strftime(format)]
|
||||
boundary = next_boundary(boundary)
|
||||
end
|
||||
rows
|
||||
end
|
||||
|
||||
def trend
|
||||
points = periods
|
||||
metadata = { count: points.length, group_count: points.length, state_rows: [],
|
||||
from: @from.iso8601, before: @before.iso8601, grain: @grain }
|
||||
return metadata if points.empty?
|
||||
|
||||
values = points.map do |boundary, period|
|
||||
"SELECT CAST(#{ quote(boundary) } AS DATETIME(6)) AS boundary, " +
|
||||
"#{ quote(period) } AS period"
|
||||
end.join(' UNION ALL ')
|
||||
order = order_sql({ 'period' => 'boundary', 'count' => 'post_count' },
|
||||
'period:asc', ['boundary ASC'])
|
||||
known = @post_start ? "periods.boundary > #{ quote(@post_start) }" : 'FALSE'
|
||||
rows = @connection.select_all(<<~SQL)
|
||||
WITH periods AS (#{ values }), series AS (
|
||||
SELECT periods.boundary, periods.period,
|
||||
CASE WHEN #{ known } THEN COUNT(DISTINCT p.post_id) ELSE NULL END AS post_count
|
||||
FROM periods
|
||||
LEFT JOIN post_versions p ON p.event_type <> 'discard'
|
||||
AND #{ membership('p') }
|
||||
AND #{ latest_condition('post_versions', 'p', 'periods.boundary') }
|
||||
AND #{ tag_visible('periods.boundary') }
|
||||
GROUP BY periods.boundary, periods.period
|
||||
)
|
||||
SELECT * FROM series ORDER BY #{ order }
|
||||
LIMIT #{ @limit } OFFSET #{ (@page - 1) * @limit }
|
||||
SQL
|
||||
metadata.merge(state_rows: rows.map do |row|
|
||||
{ kind: @kind, tag_id: @tag, name: nil, period: row['period'],
|
||||
at: row['boundary'].utc.iso8601(6), count: row['post_count']&.to_i }
|
||||
end)
|
||||
end
|
||||
|
||||
def order_sql(expressions, default, final)
|
||||
raw = @params.fetch('sort', default)
|
||||
sorts = raw.empty? ? [] : QueryState.list(raw).map { _1.split(':', -1) }
|
||||
raise ArgumentError if sorts.map(&:first).uniq.length != sorts.length
|
||||
raise ArgumentError if sorts.length > expressions.length
|
||||
|
||||
order = sorts.flat_map do |key, direction, *extra|
|
||||
if extra.any? || !expressions.key?(key) || !['asc', 'desc'].include?(direction)
|
||||
raise ArgumentError
|
||||
end
|
||||
expression = expressions.fetch(key)
|
||||
["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"]
|
||||
end
|
||||
(order + final).join(', ')
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,348 @@
|
||||
require 'csv'
|
||||
|
||||
# Export uses the exact same population and grouped cells as the interactive table.
|
||||
class CountsTableExport
|
||||
EXTRA = ['format', 'columns', 'headers', 'labels', 'children', 'child_sort',
|
||||
'order', 'nested_json', 'summary_columns', 'basis', 'axis', 'latest'].freeze
|
||||
attr_reader :format, :resource
|
||||
|
||||
def initialize(params)
|
||||
params = CountsResourceDefinition.normalise_params(params)
|
||||
QueryState.validate!(params, CountsTableQuery::KEYS + EXTRA + ['measures'])
|
||||
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
|
||||
|
||||
params = CountsDefaults.for(params).merge(params)
|
||||
unless params.key?('children')
|
||||
params = params.merge('children' => CountsExportProjection.default_children(
|
||||
params.fetch('resource', 'posts')).join(' '))
|
||||
end
|
||||
@params = params
|
||||
@labels = JSON.parse(params.fetch('labels', '{}'))
|
||||
raise ArgumentError unless @labels.is_a?(Hash)
|
||||
@selected = params.key?('children') ? QueryState.list(params['children']) : nil
|
||||
@order = QueryState.list(params.fetch('order', ''))
|
||||
@child_sort = QueryState.list(params.fetch('child_sort', '')).map { _1.split(':', -1) }
|
||||
@child_sort.each do |path, direction, *extra|
|
||||
raise ArgumentError if extra.any? || !['asc', 'desc'].include?(direction)
|
||||
raise ArgumentError unless path&.match?(/\A[a-zA-Z_]+(?:\.[a-zA-Z_]+)*\/[a-zA-Z_]+\z/)
|
||||
end
|
||||
@format = params.fetch('format', 'json')
|
||||
@resource = params.fetch('resource', 'posts')
|
||||
raise ArgumentError unless ['json', 'csv'].include?(format)
|
||||
|
||||
@query = CountsTableQuery.new(params.select { |key, _value| CountsTableQuery::KEYS.include?(key) ||
|
||||
key == 'measures' || QueryState.valid_key?(key) }
|
||||
.merge('page' => '1', 'limit' => '100'))
|
||||
@summary = @query.mode == 'summary'
|
||||
default_columns = @summary ? @query.axes.all.map { _1.fetch('field') } + ['count'] :
|
||||
CountsExportProjection.default_columns(resource)
|
||||
@columns = QueryState.list(params.fetch('columns', default_columns.join(' ')))
|
||||
default_headers = @columns.map { @labels.fetch(_1, _1) }
|
||||
@headers = JSON.parse(params.fetch('headers', default_headers.to_json))
|
||||
raise ArgumentError unless @headers.is_a?(Array) && @headers.all? { _1.is_a?(String) }
|
||||
if @summary && @headers.any? && @headers.length != @query.axes.all.length + 1
|
||||
raise ArgumentError
|
||||
end
|
||||
|
||||
unless @summary
|
||||
if ['posts', 'tags'].include?(resource)
|
||||
@projection = CountsExportProjection.new(params, resource:, summary: false, tags: false)
|
||||
else
|
||||
allowed = @query.population.definition.fields.keys + ['latest']
|
||||
raise ArgumentError if (@columns - allowed).any?
|
||||
raise ArgumentError unless @columns.length == @headers.length && @columns.uniq == @columns
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def body(progress: nil, phase: nil)
|
||||
Enumerator.new do |output|
|
||||
ApplicationRecord.transaction(isolation: :repeatable_read) do
|
||||
processed = 0
|
||||
started = false
|
||||
first = true
|
||||
@query.each_page(before_page: -> { phase&.call('aggregating') }) do |page|
|
||||
total = @summary ? nil : page[:count]
|
||||
progress&.call(processed, total)
|
||||
phase&.call('writing')
|
||||
unless started
|
||||
metadata = page.except(:versions, :cells, :axis_labels, :axis_names, :column_values,
|
||||
:row_headers, :column_headers, :pivot_totals).merge(
|
||||
resource:, record_unit: @summary && format == 'json' ? 'pivot_cell' : 'row',
|
||||
body_row_totals: 'once_per_logical_group', page_grand_total_repeated: false,
|
||||
logical_count_unit: 'regular_result_rows',
|
||||
name_collection: 'all_recorded_languages',
|
||||
name_columns: @query.population.definition.name_columns,
|
||||
aggregate_records_in_logical_count: false,
|
||||
filters: @params.select { |key, _value| CountsTableQuery::KEYS.include?(key) ||
|
||||
key == 'measures' || QueryState.valid_key?(key) },
|
||||
fields: @query.population.definition.metadata,
|
||||
columns: @columns, column_labels: @headers,
|
||||
order: @params['order'], children: @params['children'],
|
||||
sort: @params.fetch('sort', ''), child_sort: @params['child_sort'])
|
||||
output << (format == 'json' ? "{\"metadata\":#{ metadata.to_json },\"data\":[" : "\uFEFF")
|
||||
if format == 'csv'
|
||||
headings = @summary ? pivot_headers(page) : [@headers]
|
||||
headings.each do |headers|
|
||||
output << CSV.generate_line(headers.map { safe(_1) }, row_sep: "\r\n")
|
||||
end
|
||||
end
|
||||
started = true
|
||||
end
|
||||
if @summary
|
||||
pivot_rows(page).each do |header|
|
||||
records = page.fetch(:column_headers).map { pivot_cell(header, _1, page) }
|
||||
if format == 'csv'
|
||||
labels = pivot_row_labels(header, page)
|
||||
values = labels + records.map { _1[:count] }
|
||||
output << CSV.generate_line(values.map { safe(_1) }, row_sep: "\r\n")
|
||||
processed += 1
|
||||
progress&.call(processed, nil)
|
||||
else
|
||||
records.each do |record|
|
||||
output << ',' unless first
|
||||
output << record.to_json
|
||||
first = false
|
||||
processed += 1
|
||||
progress&.call(processed, nil)
|
||||
end
|
||||
end
|
||||
end
|
||||
next
|
||||
end
|
||||
page[:versions].each do |row|
|
||||
record = project(row)
|
||||
if format == 'json'
|
||||
output << ',' unless first
|
||||
output << record.to_json
|
||||
else
|
||||
values = if @summary
|
||||
row[:values].each_with_index.map do |value, index|
|
||||
summary_value(value, index, page)
|
||||
end + [row[:count]]
|
||||
elsif @projection
|
||||
@projection.csv_record(record, row)
|
||||
else
|
||||
record.map { |key, value| csv_logical(key, value) }
|
||||
end
|
||||
output << CSV.generate_line(values.map { safe(_1) }, row_sep: "\r\n")
|
||||
end
|
||||
first = false
|
||||
processed += 1
|
||||
progress&.call(processed, total)
|
||||
end
|
||||
end
|
||||
progress&.call(processed, processed)
|
||||
output << ']}' if format == 'json'
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def pivot_rows(page)
|
||||
page.fetch(:row_headers).reject do |header|
|
||||
header[:kind] == 'grand_total' && @query.page * @query.limit < page[:group_count]
|
||||
end
|
||||
end
|
||||
|
||||
def total_label(header)
|
||||
key = header[:label_key] || (header[:kind] == 'grand_total' ?
|
||||
(@query.axes.all.length == 1 ? 'singleTotal' : 'grandTotal') : 'subtotal')
|
||||
@labels.fetch(key, '')
|
||||
end
|
||||
|
||||
def pivot_row_labels(header, page)
|
||||
values = header[:values].each_with_index.map { |value, index|
|
||||
summary_value(value, index, page)
|
||||
}
|
||||
if @previous_pivot_row
|
||||
values.each_index do |index|
|
||||
if (@previous_pivot_row[:values].length > index &&
|
||||
header[:values].first(index + 1) == @previous_pivot_row[:values].first(index + 1))
|
||||
values[index] = ''
|
||||
end
|
||||
end
|
||||
end
|
||||
@previous_pivot_row = header
|
||||
values << total_label(header) unless header[:kind] == 'data'
|
||||
leading = [@query.axes.rows.length, @query.axes.columns.any? ? 1 : 0].max
|
||||
Array.new(leading) { values[_1] || '' }
|
||||
end
|
||||
|
||||
def pivot_headers(page)
|
||||
row_count = @query.axes.rows.length
|
||||
cols = @query.axes.columns
|
||||
leading = [row_count, cols.any? ? 1 : 0].max
|
||||
columns = page.fetch(:column_headers)
|
||||
headings = Array.new([cols.length, 1].max) do |level|
|
||||
prefix = Array.new(leading) { level.zero? ? @headers[_1].to_s : '' }
|
||||
prefix = [''] if row_count.zero? && cols.any?
|
||||
prefix + columns.each_with_index.map do |header, index|
|
||||
values = header[:values]
|
||||
if cols.empty?
|
||||
@labels.fetch('count', '')
|
||||
elsif level == values.length && header[:kind] != 'data'
|
||||
total_label(header)
|
||||
elsif level < values.length
|
||||
previous = index.positive? ? columns[index - 1][:values] : nil
|
||||
if level < cols.length - 1 && previous && previous.first(level + 1) == values.first(level + 1)
|
||||
''
|
||||
else
|
||||
summary_value(values[level], row_count + level, page)
|
||||
end
|
||||
else
|
||||
''
|
||||
end
|
||||
end
|
||||
end
|
||||
return headings if cols.empty?
|
||||
|
||||
cols.each_index.flat_map do |level|
|
||||
prefix = Array.new(leading, '')
|
||||
prefix = headings.first.first(leading) if level.zero?
|
||||
names = columns.each_with_index.map do |header, index|
|
||||
next '' if header[:values].length < level
|
||||
|
||||
previous = index.positive? ? columns[index - 1][:values] : nil
|
||||
if (previous && previous.length >= level &&
|
||||
previous.first(level) == header[:values].first(level))
|
||||
''
|
||||
else
|
||||
@headers[row_count + level].to_s
|
||||
end
|
||||
end
|
||||
headings[level] = Array.new(leading, '') + headings[level].drop(leading) if level.zero?
|
||||
[prefix + names, headings[level]]
|
||||
end
|
||||
end
|
||||
|
||||
def pivot_cell(row, column, page)
|
||||
@pivot_lookup ||= {}
|
||||
# Discard the previous page's lookup; memory remains bounded by one page and its rollups.
|
||||
unless @pivot_page.equal?(page)
|
||||
@pivot_page = page
|
||||
@pivot_lookup = page.fetch(:cells).to_h do |cell|
|
||||
values = cell[:values]
|
||||
[[values.first(@query.axes.rows.length), values.drop(@query.axes.rows.length)], cell[:count]]
|
||||
end
|
||||
page.fetch(:pivot_totals).each { @pivot_lookup[[_1[:row], _1[:column]]] = _1[:count] }
|
||||
end
|
||||
kinds = [row[:kind], column[:kind]]
|
||||
kind = if kinds.include?('grand_total')
|
||||
'grand_total'
|
||||
elsif kinds.include?('subtotal')
|
||||
'subtotal'
|
||||
else
|
||||
'data'
|
||||
end
|
||||
values = row[:values].each_with_index.to_h { |value, index| ["a#{ index }", value] }
|
||||
column[:values].each_with_index do |value, index|
|
||||
values["a#{ @query.axes.rows.length + index }"] = value
|
||||
end
|
||||
names = page.fetch(:axis_names, {}).to_h do |key, entries|
|
||||
[key, entries.fetch(values[key].to_s, {})]
|
||||
end
|
||||
{ row: row[:values], column: column[:values], kind:,
|
||||
axis_names: names,
|
||||
row_kind: row[:kind], column_kind: column[:kind],
|
||||
row_depth: row[:values].length, column_depth: column[:values].length,
|
||||
label_key: CountsPivotLayout.label_key(@query.axes.rows.length, @query.axes.columns.length,
|
||||
row[:values].length, column[:values].length),
|
||||
count: @pivot_lookup.fetch([row[:values], column[:values]], 0) }
|
||||
end
|
||||
|
||||
def summary_value(value, index, page)
|
||||
field = @query.axes.all.fetch(index).fetch('field')
|
||||
return @labels.fetch('bot', '') if field == 'user' && value.nil?
|
||||
if ['language_code', 'source_language_code'].include?(field)
|
||||
return @labels.fetch("language:#{ value }", value)
|
||||
end
|
||||
|
||||
names = page[:axis_labels].fetch("a#{ index }", {})
|
||||
name = field == 'user' ? names[value.to_s] : names.fetch(value.to_s, value)
|
||||
if field == 'user'
|
||||
name = @labels.fetch('notRegistered', '') if name.nil? || name == ''
|
||||
pattern = @labels.fetch('actorFormat', '{{name}} (#{{id}})')
|
||||
return pattern.gsub(/\{\{(name|id)\}\}/) do
|
||||
Regexp.last_match(1) == 'name' ? name.to_s : value.to_s
|
||||
end
|
||||
end
|
||||
@labels.fetch("#{ field }:#{ value }", name)
|
||||
end
|
||||
|
||||
def project(row)
|
||||
return { row: row[:values].first(@query.axes.rows.length),
|
||||
column: row[:values].drop(@query.axes.rows.length), count: row[:count] } if @summary
|
||||
return @projection.project(row) if @projection
|
||||
|
||||
@columns.to_h do |column|
|
||||
key = column == 'user' ? 'created_by_user' : column
|
||||
value = row[key]
|
||||
value = project_nested(value, key) if ['members', 'placements', 'export_paths_json'].include?(key)
|
||||
[key, value]
|
||||
end
|
||||
end
|
||||
|
||||
def camel(key)
|
||||
key.gsub(/_([a-z])/) { Regexp.last_match(1).upcase }
|
||||
end
|
||||
|
||||
def project_nested(value, path)
|
||||
rows = path == 'export_paths_json' ? (value || {}).map { |profile, target|
|
||||
{ 'profile' => profile, 'path' => target }
|
||||
} : Array(value)
|
||||
sorts = @child_sort.filter_map do |key, direction|
|
||||
parent, field = key.split('/', 2)
|
||||
[field, direction] if parent == path
|
||||
end
|
||||
rows = rows.each_with_index.sort do |(first, a), (last, b)|
|
||||
compared = 0
|
||||
sorts.each do |field, direction|
|
||||
left = first.find { |key, _| camel(key) == field }&.last
|
||||
right = last.find { |key, _| camel(key) == field }&.last
|
||||
compared = if left.nil? || right.nil?
|
||||
left.nil? == right.nil? ? 0 : (left.nil? ? 1 : -1)
|
||||
else
|
||||
(left.to_s <=> right.to_s) * (direction == 'asc' ? 1 : -1)
|
||||
end
|
||||
break unless compared.zero?
|
||||
end
|
||||
compared.zero? ? a <=> b : compared
|
||||
end.map(&:first)
|
||||
rows.map do |row|
|
||||
fields = row.keys.select { !@selected || @selected.include?("#{ path }.#{ camel(_1) }") }
|
||||
ordered = (@order + fields.map { "#{ path }.#{ camel(_1) }" }).uniq
|
||||
.filter_map { |item| fields.find { "#{ path }.#{ camel(_1) }" == item } }
|
||||
ordered.to_h do |key|
|
||||
item = row[key]
|
||||
item = project_nested(item, "#{ path }.#{ camel(key) }") if item.is_a?(Array)
|
||||
[key, item]
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def csv_logical(key, value)
|
||||
return @labels.fetch('bot', '') if key == 'created_by_user' && value.nil?
|
||||
return value ? @labels.fetch('latest', '') : '' if key == 'latest'
|
||||
if value.is_a?(Time) || value.is_a?(ActiveSupport::TimeWithZone)
|
||||
return @query.population.timezone.utc_to_local(value.utc).iso8601
|
||||
end
|
||||
if key == 'language_code' || key == 'source_language_code'
|
||||
return @labels.fetch("language:#{ value }", value)
|
||||
end
|
||||
if value.is_a?(Hash)
|
||||
return value.to_h { |field, item| [@labels.fetch(camel(field.to_s), ''), csv_logical(field, item)] }
|
||||
end
|
||||
return value.map { csv_logical(key, _1) } if value.is_a?(Array)
|
||||
|
||||
@labels.fetch("#{ key }:#{ value }", value)
|
||||
end
|
||||
|
||||
def safe(value)
|
||||
return value if value.is_a?(Numeric)
|
||||
|
||||
text = value.is_a?(Hash) || value.is_a?(Array) ? JSON.generate(value) : value.to_s
|
||||
text.match?(/\A\s*[=+@-]/) ? "'#{ text }" : text
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,497 @@
|
||||
# Common entrypoint for six version resources; legacy endpoints keep their contracts.
|
||||
class CountsTableQuery
|
||||
MAX_COLUMNS = 100
|
||||
class TooManyColumns < ArgumentError; end
|
||||
|
||||
KEYS = ['resource', 'definition', 'mode', 'at', 'execution_at', 'from', 'through',
|
||||
'timezone', 'rows', 'cols', 'conditions', 'drill', 'tags', 'tag_refs', 'match',
|
||||
'tag', 'kind', 'resource_id', 'user_id', 'page', 'limit', 'sort', 'locale',
|
||||
'measures', 'column_order', 'condition_values', 'name_scope'].freeze
|
||||
attr_reader :population, :axes, :page, :limit, :mode
|
||||
|
||||
def initialize(params)
|
||||
params = CountsResourceDefinition.normalise_params(params)
|
||||
QueryState.validate!(params, KEYS + ['measures'])
|
||||
raise ArgumentError unless params.values.all? { _1.is_a?(String) }
|
||||
|
||||
params = CountsDefaults.for(params).slice(*KEYS).merge(params)
|
||||
@params = params
|
||||
@mode = params.fetch('mode', 'summary')
|
||||
validate_summary_conditions!
|
||||
@population = CountsPopulation.new(params, authorisation_flg: mode != 'summary')
|
||||
@definition = population.definition
|
||||
@connection = @definition.model.connection
|
||||
@axes = CountsAxes.new(population)
|
||||
@page = Integer(params.fetch('page', '1'), 10)
|
||||
@limit = Integer(params.fetch('limit', '100'), 10)
|
||||
@mode = params.fetch('mode', 'summary')
|
||||
raise ArgumentError unless page.between?(1, 1_000_000) && limit.between?(1, 100)
|
||||
raise ArgumentError unless ['summary', 'details'].include?(mode)
|
||||
|
||||
@sort = QueryState.list(params.fetch('sort', '')).map { _1.split(':', -1) }
|
||||
allowed = mode == 'summary' ? axes.all.each_index.map { "a#{ _1 }" } + ['count'] :
|
||||
@definition.fields.keys + ['latest'] +
|
||||
(@definition.resource == 'tags' ? @definition.name_columns.map { _1[:key] } : []) +
|
||||
(@definition.resource == 'posts' ? ['tags', 'parents'] : []) +
|
||||
(@definition.resource == 'tags' ? ['parents', 'deprecated'] : [])
|
||||
raise ArgumentError if @sort.map(&:first).uniq.length != @sort.length
|
||||
@sort.each do |key, direction, *extra|
|
||||
measure = mode == 'summary' && key&.match?(/\Am[0-9]{1,2}\z/)
|
||||
measure_values(key) if measure
|
||||
if extra.any? || (!allowed.include?(key) && !measure) || !['asc', 'desc'].include?(direction)
|
||||
raise ArgumentError
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def call
|
||||
ApplicationRecord.transaction do
|
||||
result = metadata.merge(mode == 'summary' ? summary : details)
|
||||
visible = mode == 'summary' ? result[:visible_output_count] : result[:count]
|
||||
totals = output_totals(visible)
|
||||
result.merge(visible_output_count: visible,
|
||||
**totals,
|
||||
output_count_scope: 'before_authorisation',
|
||||
output_count_unit: 'regular_result_rows',
|
||||
page_grand_total_repeated: mode == 'summary' && axes.rows.any? &&
|
||||
result[:group_count] > limit)
|
||||
end
|
||||
end
|
||||
|
||||
def metadata
|
||||
{ condition_labels: condition_labels, population_count: nil, population_count_computed: false,
|
||||
excluded_count: nil, excluded_count_scope: 'current_query',
|
||||
population_count_scope: 'all_recorded_versions',
|
||||
page:, limit:, execution_at: population.at.iso8601(6), at: @params['at'].presence,
|
||||
definition: population.history ? 'history' : 'latest',
|
||||
timezone: population.timezone.identifier, rows: axes.rows, cols: axes.columns,
|
||||
fields: @definition.metadata,
|
||||
masked_value: HistoryAccess.admin? ? nil : CountsAxes::MASKED_VALUE,
|
||||
count: record_counts[:visible] }
|
||||
end
|
||||
|
||||
def summary
|
||||
if axes.all.empty?
|
||||
headers = [{ values: [], kind: 'data', depth: 0, axis_count: 0, label_key: nil }]
|
||||
return { group_count: 1, visible_output_count: 1, column_values: [[]],
|
||||
row_headers: page == 1 ? headers : [], column_headers: headers, pivot_totals: [],
|
||||
cells: page == 1 ? [{ values: [], count: record_counts[:visible] }] : [],
|
||||
axis_labels: {} }
|
||||
end
|
||||
expressions = axes.expressions
|
||||
selected = expressions.each_with_index.map do |expression, index|
|
||||
"#{ expression } AS a#{ index }"
|
||||
end
|
||||
source = summary_population.scope
|
||||
source = source.joins(axes.joins) unless axes.joins.empty?
|
||||
sql = if expressions.empty?
|
||||
source.except(:select).select('COUNT(DISTINCT v.id) AS count').to_sql
|
||||
else
|
||||
source.except(:select).group(*expressions.map { Arel.sql(_1) })
|
||||
.select(Arel.sql((selected + ['COUNT(DISTINCT v.id) AS count']).join(', ')))
|
||||
.to_sql
|
||||
end
|
||||
prefix = "WITH cells AS (#{ sql })"
|
||||
column_values = limited_columns(prefix)
|
||||
row_keys = axes.rows.each_index.map { "a#{ _1 }" }
|
||||
# Page whole row tuples so a pivot row is never split across display pages.
|
||||
row_selection = row_keys.empty? ? '1 AS singleton' : row_keys.join(', ')
|
||||
row_group = row_keys.empty? ? 'GROUP BY singleton' : "GROUP BY #{ row_keys.join(', ') }"
|
||||
measure_joins = []
|
||||
extras = []
|
||||
row_sort = @sort.each_with_index.filter_map do |(key, direction), index|
|
||||
if key == 'count' || key.match?(/\Am[0-9]+\z/)
|
||||
values = key == 'count' ? [] : measure_values(key)
|
||||
relation = measure_relation(values)
|
||||
matches = row_keys.map { "grouped.#{ _1 } <=> sort#{ index }.#{ _1 }" }
|
||||
condition = matches.empty? ? 'TRUE' : matches.join(' AND ')
|
||||
measure_joins << "LEFT JOIN (#{ relation }) sort#{ index } ON #{ condition }"
|
||||
extras << "sort#{ index }.count AS sort#{ index }"
|
||||
"sort#{ index } IS NULL ASC, sort#{ index } #{ direction.upcase }"
|
||||
elsif row_keys.include?(key)
|
||||
axis_order_sql(key, direction)
|
||||
end
|
||||
end
|
||||
order_rows = row_sort + row_keys.map { axis_order_sql(_1) }
|
||||
order_rows = ['singleton ASC'] if order_rows.empty?
|
||||
grouped = "SELECT #{ row_selection } FROM cells #{ row_group }"
|
||||
selection = (['grouped.*'] + extras).join(', ')
|
||||
group_sql = "SELECT #{ selection } FROM (#{ grouped }) grouped #{ measure_joins.join(' ') }"
|
||||
join = row_keys.map { "cells.#{ _1 } <=> page_rows.#{ _1 }" }.join(' AND ')
|
||||
join = 'TRUE' if row_keys.empty?
|
||||
order_cols = @sort.filter_map do |key, direction|
|
||||
next unless key.match?(/\Aa[0-9]+\z/) && !row_keys.include?(key)
|
||||
|
||||
axis_order_sql(key, direction, qualifier: 'cells')
|
||||
end
|
||||
order = ['page_rows.position'] + order_cols +
|
||||
axes.all.each_index.map { axis_order_sql("a#{ _1 }", qualifier: 'cells') }
|
||||
result = @connection.select_all(<<~SQL).to_a
|
||||
#{ prefix }, row_groups AS (#{ group_sql }),
|
||||
numbered AS (SELECT row_groups.*,
|
||||
ROW_NUMBER() OVER (ORDER BY #{ order_rows.join(', ') }) AS position FROM row_groups),
|
||||
group_totals AS (SELECT COUNT(*) AS group_count FROM row_groups)
|
||||
SELECT /*+ MAX_EXECUTION_TIME(10000) */ cells.*,
|
||||
page_rows.position AS counts_position, group_totals.group_count AS counts_group_count
|
||||
FROM group_totals LEFT JOIN numbered page_rows
|
||||
ON page_rows.position > #{ (page - 1) * limit }
|
||||
AND page_rows.position <= #{ page * limit + 1 }
|
||||
LEFT JOIN cells ON #{ join }
|
||||
ORDER BY #{ order.join(', ') }
|
||||
SQL
|
||||
total = result.first.fetch('counts_group_count').to_i
|
||||
next_row = result.find { _1['counts_position'].to_i == page * limit + 1 }
|
||||
records = result.select do |record|
|
||||
record['counts_position'] && record['counts_position'].to_i <= page * limit
|
||||
end
|
||||
row_values = records.map { |record| row_keys.map { record[_1] } }.uniq
|
||||
row_depths = CountsPivotLayout.depths(row_keys, @sort)
|
||||
column_keys = axes.columns.each_index.map { "a#{ axes.rows.length + _1 }" }
|
||||
column_sort = @sort
|
||||
if @params.key?('column_order') && column_keys.any? &&
|
||||
!(@sort.first&.first == 'count' || @sort.first&.first&.match?(/\Am[0-9]+\z/))
|
||||
column_sort = [[column_keys.first, 'asc']] + @sort
|
||||
end
|
||||
column_depths = CountsPivotLayout.depths(column_keys, column_sort, measures: false)
|
||||
row_headers = CountsPivotLayout.headers(row_values, row_depths,
|
||||
next_values: next_row && row_keys.map { next_row[_1] }, grand: axes.rows.any?,
|
||||
axis_count: axes.rows.length, other_count: axes.columns.length)
|
||||
if axes.rows.empty? && axes.columns.any?
|
||||
row_headers = CountsPivotLayout.headers([[]], [], axis_count: 0,
|
||||
other_count: axes.columns.length).map { _1.merge(kind: 'grand_total') }
|
||||
end
|
||||
column_headers = CountsPivotLayout.headers(column_values, column_depths,
|
||||
grand: axes.columns.any?, axis_count: axes.columns.length,
|
||||
other_count: axes.rows.length, direction: :column)
|
||||
if axes.columns.empty? && axes.rows.any?
|
||||
column_headers = CountsPivotLayout.headers([[]], [], axis_count: 0,
|
||||
other_count: axes.rows.length, direction: :column)
|
||||
.map { _1.merge(kind: 'grand_total') }
|
||||
end
|
||||
totals = CountsPivotTotals.new(summary_population, axes, row_headers, column_headers).call
|
||||
{ group_count: total,
|
||||
visible_output_count: axes.rows.empty? ? 1 : total,
|
||||
column_values:, row_headers:, column_headers:, pivot_totals: totals,
|
||||
cells: records.map do |row|
|
||||
{ values: axes.all.each_index.map { row["a#{ _1 }"] }, count: row['count'].to_i }
|
||||
end, axis_labels: axis_labels(records, column_values), axis_names: @axis_names }
|
||||
end
|
||||
|
||||
def details
|
||||
query = population.scope.select(Arel.sql(
|
||||
"(#{ population.latest_sql }) AS counts_latest"))
|
||||
order = sort_sql { detail_expression(_1) }
|
||||
query = query.order(Arel.sql((order + ['v.id ASC']).join(', ')))
|
||||
.limit(limit).offset((page - 1) * limit)
|
||||
rows = query.to_a
|
||||
if ['posts', 'tags'].include?(@definition.resource)
|
||||
point = population.history ? population.at :
|
||||
population.timestamp(@params['at']) || population.at
|
||||
values = CountsVersionRepr.many(rows, resource: @definition.resource,
|
||||
language_code: locale.language_code,
|
||||
at: point, history_flg: population.history)
|
||||
values.each_with_index { |value, index| value[:event_type] = rows[index].event_type }
|
||||
else
|
||||
users = User.where(id: rows.map(&:created_by_user_id)).pluck(:id, :name).to_h
|
||||
values = rows.map do |row|
|
||||
attributes = @definition.fields.keys.to_h do |key|
|
||||
[key, row.public_send(@definition.column(key))]
|
||||
end
|
||||
attributes.merge('id' => row.id, 'resource_id' => row[@definition.keys.first],
|
||||
'latest' => row.attributes['counts_latest'].to_i == 1,
|
||||
'created_by_user' => row.created_by_user_id &&
|
||||
{ id: row.created_by_user_id, name: users[row.created_by_user_id] })
|
||||
end
|
||||
end
|
||||
{ versions: values }
|
||||
end
|
||||
|
||||
def each_page(before_page: nil)
|
||||
total = nil
|
||||
loop do
|
||||
before_page&.call
|
||||
result = call
|
||||
total ||= mode == 'summary' ? result[:group_count] : result[:count]
|
||||
yield result
|
||||
break if page * limit >= total
|
||||
|
||||
@page += 1
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# Public timeline predicates may include hidden records. Private predicates must
|
||||
# never become a count oracle, even when the selected axes themselves are public.
|
||||
def validate_summary_conditions!
|
||||
return if mode != 'summary' || HistoryAccess.admin?
|
||||
|
||||
fields = QueryState.records(@params, 'conditions') + QueryState.records(@params, 'drill')
|
||||
private_fields = fields.any? { !CountsAxes::PUBLIC_FIELDS.include?(_1['field']) }
|
||||
private_parameters = ['tags', 'tag_refs', 'tag', 'kind', 'resource_id', 'user_id',
|
||||
'name_scope'].any? do |key|
|
||||
@params.any? { |name, value| (name == key || name.start_with?("#{ key }.")) && value != '' }
|
||||
end
|
||||
raise CountsAxes::PrivateCondition if private_fields || private_parameters
|
||||
|
||||
['column_order', 'measures'].each do |key|
|
||||
next if QueryState.records(@params, key).empty?
|
||||
columns = QueryState.axes(@params.fetch('cols', ''))
|
||||
raise CountsAxes::PrivateCondition if columns.any? do |axis|
|
||||
!CountsAxes::PUBLIC_FIELDS.include?(axis['field'])
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def unrestricted_population
|
||||
@unrestricted_population ||= CountsPopulation.new(
|
||||
@params.merge('execution_at' => population.at.iso8601(6)), authorisation_flg: false)
|
||||
end
|
||||
|
||||
def summary_population
|
||||
population
|
||||
end
|
||||
|
||||
def record_counts
|
||||
return @record_counts if @record_counts
|
||||
|
||||
if HistoryAccess.admin? || mode == 'summary'
|
||||
total = population.scope.except(:select)
|
||||
.optimizer_hints('MAX_EXECUTION_TIME(10000)').count
|
||||
return @record_counts = { total:, visible: total, denied: 0 }
|
||||
end
|
||||
total, denied = unrestricted_population.scope.except(:select)
|
||||
.optimizer_hints('MAX_EXECUTION_TIME(10000)').pick(Arel.sql('COUNT(*)'),
|
||||
Arel.sql("COALESCE(SUM(CASE WHEN #{ HistoryAccess.sql(@definition.model, 'v') } " +
|
||||
'THEN 0 ELSE 1 END), 0)'))
|
||||
# Related/previous-version predicates may differ under public authority.
|
||||
shared = QueryState.records(@params, 'conditions').empty? &&
|
||||
QueryState.records(@params, 'drill').none? { _1['field'] == 'changed_field' }
|
||||
visible = if shared
|
||||
total.to_i - denied.to_i
|
||||
else
|
||||
population.scope.except(:select)
|
||||
.optimizer_hints('MAX_EXECUTION_TIME(10000)').count
|
||||
end
|
||||
@record_counts = { total: total.to_i, denied: denied.to_i, visible: }
|
||||
end
|
||||
|
||||
def output_totals visible_rows
|
||||
return @output_totals if @output_totals
|
||||
|
||||
if mode == 'summary'
|
||||
return @output_totals = { output_count: visible_rows, excluded_count: nil }
|
||||
end
|
||||
if HistoryAccess.admin?
|
||||
return @output_totals = { output_count: visible_rows, excluded_count: 0 }
|
||||
end
|
||||
# Only numerical aggregates leave this population. Labels, snapshots and
|
||||
# detail pagination continue to use the authorised population exclusively.
|
||||
source = unrestricted_population
|
||||
rows = if mode == 'details'
|
||||
record_counts[:total]
|
||||
else
|
||||
summary_output_count(source)
|
||||
end
|
||||
@output_totals = { output_count: rows, excluded_count: record_counts[:denied] }
|
||||
end
|
||||
|
||||
def summary_output_count(source)
|
||||
keys = axes.rows.each_index.map { "a#{ _1 }" }
|
||||
return 1 if keys.empty?
|
||||
|
||||
unrestricted_axes = CountsAxes.new(source)
|
||||
relation = source.scope
|
||||
joins = unrestricted_axes.joins
|
||||
relation = relation.joins(joins) unless joins.empty?
|
||||
expressions = unrestricted_axes.expressions
|
||||
selected = expressions.each_with_index.map { |expression, index|
|
||||
"#{ expression } AS a#{ index }"
|
||||
}
|
||||
sql = relation.except(:select).select(Arel.sql(selected.join(', ')))
|
||||
.distinct.to_sql
|
||||
prefix = "WITH cells AS (#{ sql })"
|
||||
@connection.select_value(<<~SQL).to_i
|
||||
#{ prefix }
|
||||
SELECT /*+ MAX_EXECUTION_TIME(10000) */ COUNT(*) FROM (
|
||||
SELECT #{ keys.join(', ') } FROM cells GROUP BY #{ keys.join(', ') }
|
||||
) output_groups
|
||||
SQL
|
||||
end
|
||||
|
||||
def locale = @locale ||= Locale.find_by!(code: @params.fetch('locale', 'ja'))
|
||||
|
||||
def limited_columns prefix
|
||||
keys = axes.columns.each_index.map { "a#{ axes.rows.length + _1 }" }
|
||||
return [[]] if keys.empty?
|
||||
|
||||
configured = @sort.select { |key, _direction| keys.include?(key) }
|
||||
directions = (configured + keys.map { [_1, 'asc'] }).uniq { _1.first }
|
||||
order = directions.map { |key, direction| axis_order_sql(key, direction) }.join(', ')
|
||||
custom = QueryState.records(@params, 'column_order')
|
||||
if custom.any?
|
||||
raise ArgumentError unless custom.map { _1['value'] }.uniq.length == custom.length
|
||||
|
||||
cases = custom.each_with_index.map do |record, index|
|
||||
"WHEN #{ keys.first } <=> #{ @connection.quote(record['value']) } THEN #{ index }"
|
||||
end.join(' ')
|
||||
order = "CASE #{ cases } ELSE #{ custom.length } END, #{ order }"
|
||||
end
|
||||
values = @connection.select_rows(<<~SQL)
|
||||
#{ prefix }
|
||||
SELECT /*+ MAX_EXECUTION_TIME(10000) */ DISTINCT #{ keys.join(', ') } FROM cells
|
||||
ORDER BY #{ order } LIMIT #{ MAX_COLUMNS + 1 }
|
||||
SQL
|
||||
raise TooManyColumns if values.length > MAX_COLUMNS
|
||||
|
||||
values
|
||||
end
|
||||
|
||||
def measure_relation values
|
||||
row_expressions = axes.expressions.first(axes.rows.length)
|
||||
source = summary_population.scope.except(:select, :order)
|
||||
fields = axes.rows + axes.columns.first(values.length)
|
||||
joins = axes.joins(fields.map { _1.fetch('field') })
|
||||
source = source.joins(joins) unless joins.empty?
|
||||
values.each_with_index do |value, index|
|
||||
expression = axes.expressions[axes.rows.length + index]
|
||||
source = source.where("(#{ expression }) <=> ?", value)
|
||||
end
|
||||
selected = row_expressions.each_with_index.map { |expression, index|
|
||||
"#{ expression } AS a#{ index }"
|
||||
} + ['COUNT(DISTINCT v.id) AS count']
|
||||
source = source.group(*row_expressions.map { Arel.sql(_1) }) if row_expressions.any?
|
||||
source.select(Arel.sql(selected.join(', '))).to_sql
|
||||
end
|
||||
|
||||
def axis_order_sql key, direction = 'asc', qualifier: nil
|
||||
column = qualifier ? "#{ qualifier }.#{ key }" : key
|
||||
expression = column
|
||||
if axes.all.fetch(key.delete_prefix('a').to_i).fetch('field') == 'category'
|
||||
# Tag's enum preserves the same category order as TagsController.
|
||||
cases = Tag.categories.keys.each_with_index.map { |category, index|
|
||||
"WHEN #{ @connection.quote(category) } THEN #{ index }"
|
||||
}.join(' ')
|
||||
expression = "CASE #{ column } #{ cases } ELSE #{ Tag.categories.length } END"
|
||||
end
|
||||
"#{ column } IS NULL ASC, #{ expression } #{ direction.upcase }"
|
||||
end
|
||||
|
||||
def measure_values key
|
||||
records = QueryState.records(@params, 'measures').select { _1['key'] == key }
|
||||
raise ArgumentError if records.empty?
|
||||
raise ArgumentError unless records.map { _1['axis'] }.sort == (0...records.length).to_a
|
||||
|
||||
value = records.sort_by { _1['axis'] }.map { _1['value'] }
|
||||
unless (value.is_a?(Array) && value.length <= axes.columns.length &&
|
||||
value.all? { _1.nil? || _1.is_a?(String) || _1.is_a?(Numeric) })
|
||||
raise ArgumentError
|
||||
end
|
||||
value
|
||||
end
|
||||
|
||||
def detail_expression(key)
|
||||
return "(#{ population.latest_sql })" if key == 'latest'
|
||||
return @definition.name_expression(key) if @definition.resource == 'tags' &&
|
||||
@definition.name_column(key)
|
||||
aliases = { 'tags' => 'tags_json', 'parents' => @definition.resource == 'posts' ?
|
||||
'parent_post_ids' : 'parent_tag_ids', 'deprecated' => 'deprecated_at' }
|
||||
field = aliases.fetch(key, key)
|
||||
expression = @definition.expression(field)
|
||||
return expression unless @definition.fields[field] == :json
|
||||
|
||||
"CAST(#{ expression } AS CHAR) COLLATE utf8mb4_bin"
|
||||
end
|
||||
|
||||
def sort_sql
|
||||
@sort.flat_map do |key, direction|
|
||||
expression = yield key
|
||||
["(#{ expression }) IS NULL ASC", "#{ expression } #{ direction.upcase }"]
|
||||
end
|
||||
end
|
||||
|
||||
def condition_labels
|
||||
entries = QueryState.records(@params, 'conditions').select { _1['operator'] == 'axis_eq' }
|
||||
entries += QueryState.records(@params, 'drill')
|
||||
entries.group_by { _1['field'] }.to_h do |field, conditions|
|
||||
values = conditions.map { _1['value'] }.compact.uniq
|
||||
labels = case field
|
||||
when 'tags' then tag_labels(values)
|
||||
when 'user'
|
||||
User.where(id: values).pluck(:id, :name).to_h.transform_keys(&:to_s)
|
||||
else {}
|
||||
end
|
||||
[field, labels]
|
||||
end
|
||||
end
|
||||
|
||||
def axis_labels(records, column_values)
|
||||
result = {}
|
||||
@axis_names = {}
|
||||
axes.all.each_with_index do |axis, index|
|
||||
values = records.map { _1["a#{ index }"] }
|
||||
if index >= axes.rows.length
|
||||
values += column_values.map { _1[index - axes.rows.length] }
|
||||
end
|
||||
values = values.compact.uniq - [CountsAxes::MASKED_VALUE]
|
||||
labels = case axis['field']
|
||||
when 'user'
|
||||
User.where(id: values).pluck(:id, :name).to_h.transform_keys(&:to_s)
|
||||
when 'language_code', 'source_language_code'
|
||||
values.to_h { [_1, _1] }
|
||||
when 'tags'
|
||||
labels = tag_labels(values)
|
||||
@axis_names["a#{ index }"] = @tag_names
|
||||
labels
|
||||
else {}
|
||||
end
|
||||
result["a#{ index }"] = labels
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
def tag_labels(values)
|
||||
internal = values.grep(/\Ainternal:/).map { _1.split(':').last.to_i }
|
||||
external = values.grep(/\Aexternal:/).map { _1.split(':').last.to_i }
|
||||
point = population.history ? population.at :
|
||||
population.timestamp(@params['at']) || population.at
|
||||
labels = {}
|
||||
@tag_names = {}
|
||||
sources = [[TagVersion, internal, 'internal'], [NicoTagVersion, external, 'external']]
|
||||
sources.each do |model, ids, kind|
|
||||
next if ids.empty?
|
||||
|
||||
table = model.table_name
|
||||
scope = HistoryAccess.visible(model.where(tag_id: ids)).where('created_at <= ?', point)
|
||||
bound = @connection.quote(point)
|
||||
if population.history && @definition.resource == 'posts'
|
||||
key = kind == 'internal' ? 'tag_id' : 'external_tag_id'
|
||||
pairs = HistoryAccess.visible(population.scope, 'v').except(:select, :order).joins(
|
||||
"JOIN JSON_TABLE(v.tags_json, '$[*]' COLUMNS (tag_id BIGINT PATH '$.#{ key }', " +
|
||||
"version_no INT PATH '$.version_no' NULL ON EMPTY)) recorded_tag")
|
||||
.where('recorded_tag.tag_id IN (?)', ids).group('recorded_tag.tag_id')
|
||||
value = kind == 'internal' ? 'MAX(recorded_tag.version_no)' : 'MAX(v.created_at)'
|
||||
pairs = pairs.select(Arel.sql("recorded_tag.tag_id, #{ value } AS reference_point"))
|
||||
scope = scope.joins("JOIN (#{ pairs.to_sql }) recorded " +
|
||||
"ON recorded.tag_id = #{ table }.tag_id")
|
||||
if kind == 'internal'
|
||||
scope = scope.where("#{ table }.version_no = recorded.reference_point")
|
||||
else
|
||||
scope = scope.where("#{ table }.created_at <= recorded.reference_point")
|
||||
bound = 'recorded.reference_point'
|
||||
end
|
||||
end
|
||||
unless population.history && kind == 'internal' && @definition.resource == 'posts'
|
||||
scope = scope.where("NOT EXISTS (SELECT 1 FROM #{ table } newer " +
|
||||
"WHERE newer.tag_id = #{ table }.tag_id AND newer.created_at <= #{ bound } " +
|
||||
"AND newer.version_no > #{ table }.version_no)")
|
||||
end
|
||||
scope.each do |row|
|
||||
id = "#{ kind }:#{ row.tag_id }"
|
||||
labels[id] = row.name
|
||||
@tag_names[id] = row.respond_to?(:names) ? row.names : {}
|
||||
end
|
||||
end
|
||||
labels
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,52 @@
|
||||
# Shared literal parsing and reference resolution for current and snapshot post search.
|
||||
class PostTagSearch
|
||||
def self.parse_references(raw, literals, partial: false)
|
||||
return nil unless raw.key?('tag_refs')
|
||||
|
||||
references = QueryState.records(raw, 'tag_refs')
|
||||
valid = references.is_a?(Array) && references.all? do |reference|
|
||||
next false unless reference.is_a?(Hash) && reference['name'].is_a?(String)
|
||||
|
||||
ids = [reference['tag_id'], reference['external_tag_id']].compact
|
||||
((ids.any? || reference['missing'] == true) &&
|
||||
ids.all? { _1.is_a?(Integer) && _1.positive? } &&
|
||||
(!reference.key?('missing') || (reference['missing'] == true && ids.empty?)) &&
|
||||
(!reference.key?('negative') || [true, false].include?(reference['negative'])))
|
||||
end
|
||||
raise ArgumentError unless valid
|
||||
|
||||
if partial
|
||||
remaining = references.dup
|
||||
result = literals.map do |literal|
|
||||
index = remaining.index { _1['name'] == literal }
|
||||
index ? remaining.delete_at(index) : nil
|
||||
end
|
||||
raise ArgumentError if remaining.any?
|
||||
|
||||
return result
|
||||
end
|
||||
references.map { _1['name'] } == literals ? references : nil
|
||||
end
|
||||
|
||||
def self.literals(locale, raw, references: nil)
|
||||
raw.split.each_with_index.filter_map do |token, index|
|
||||
next if token.downcase == 'type:or'
|
||||
|
||||
reference = references&.[](index)
|
||||
name = token.sub(/\Anot:/i, '')
|
||||
name = TagName.canonicalise(locale, name).first unless reference || locale.nil?
|
||||
negative = reference&.key?('negative') ? reference['negative'] :
|
||||
token.downcase.start_with?('not:')
|
||||
{ name:, negative:, reference: }
|
||||
end
|
||||
end
|
||||
|
||||
def self.references(locale, literal)
|
||||
reference = literal[:reference]
|
||||
return [reference['tag_id'], reference['external_tag_id']] if reference
|
||||
|
||||
[TagName.where(language_code: locale.language_code, name: literal[:name])
|
||||
.distinct.pluck(:tag_id),
|
||||
ExternalTag.where("CONCAT(platform, ':', name) = ?", literal[:name]).pluck(:id)]
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,90 @@
|
||||
# Public query grammar shared with frontend/src/lib/queryState.ts.
|
||||
class QueryState
|
||||
SCHEMAS = {
|
||||
'conditions' => ['field', 'operator', 'value', 'end', 'blank', 'input', 'values', 'grain', 'value_type'],
|
||||
'condition_values' => ['condition', 'value', 'value_type'],
|
||||
'drill' => ['field', 'grain', 'value', 'value_type'],
|
||||
'tag_refs' => ['name', 'tag_id', 'external_tag_id', 'negative', 'missing'],
|
||||
'column_order' => ['value', 'value_type'],
|
||||
'measures' => ['key', 'axis', 'value', 'value_type'] }.freeze
|
||||
|
||||
def self.list(value)
|
||||
return [] if value.nil? || value == ''
|
||||
raise ArgumentError unless value.is_a?(String) && value.bytesize <= 32_768
|
||||
|
||||
values = value.split(/ /, -1)
|
||||
raise ArgumentError if values.length > 200 || values.any?(&:empty?) || values.uniq != values
|
||||
|
||||
values
|
||||
end
|
||||
|
||||
def self.valid_key?(key)
|
||||
root, index, field, extra = key.split('.', -1)
|
||||
!extra && SCHEMAS.key?(root) && index&.match?(/\A(?:0|[1-9][0-9]?)\z/) &&
|
||||
SCHEMAS[root].include?(field)
|
||||
end
|
||||
|
||||
def self.validate!(params, allowed)
|
||||
raise ArgumentError if params.length > 1000
|
||||
params.each do |key, value|
|
||||
raise ArgumentError unless allowed.include?(key) || valid_key?(key)
|
||||
raise ArgumentError unless value.is_a?(String) && value.bytesize <= 131_072
|
||||
end
|
||||
SCHEMAS.each_key { records(params, _1) }
|
||||
end
|
||||
|
||||
def self.records(params, key)
|
||||
records = {}
|
||||
params.each do |name, value|
|
||||
next unless name.start_with?("#{ key }.")
|
||||
raise ArgumentError unless params.key?(key) && valid_key?(name)
|
||||
|
||||
_root, index, field = name.split('.')
|
||||
record = records[index.to_i] ||= {}
|
||||
if ['blank', 'negative', 'missing', 'values'].include?(field)
|
||||
raise ArgumentError unless ['0', '1'].include?(value)
|
||||
|
||||
value = value == '1'
|
||||
elsif ['tag_id', 'external_tag_id', 'axis', 'condition'].include?(field)
|
||||
raise ArgumentError unless value.match?(/\A[0-9]{1,18}\z/)
|
||||
|
||||
value = value.to_i
|
||||
end
|
||||
record[field] = value
|
||||
end
|
||||
raise ArgumentError if params.key?(key) && params[key] != ''
|
||||
raise ArgumentError unless records.keys.sort == (0...records.length).to_a
|
||||
|
||||
records.sort.map do |_index, record|
|
||||
type = record.delete('value_type')
|
||||
if (['drill', 'measures', 'column_order', 'condition_values'].include?(key) ||
|
||||
(key == 'conditions' && record['operator'] == 'axis_eq'))
|
||||
raise ArgumentError unless ['null', 'string', 'number', 'boolean'].include?(type)
|
||||
|
||||
raise ArgumentError unless type == 'null' || record.key?('value')
|
||||
|
||||
value = record['value']
|
||||
record['value'] = case type
|
||||
when 'null' then nil
|
||||
when 'number'
|
||||
raise ArgumentError unless value&.match?(/\A-?[0-9]{1,18}\z/)
|
||||
Integer(value, 10)
|
||||
when 'boolean'
|
||||
raise ArgumentError unless ['true', 'false'].include?(value)
|
||||
value == 'true'
|
||||
else value
|
||||
end
|
||||
end
|
||||
record
|
||||
end
|
||||
end
|
||||
|
||||
def self.axes(value)
|
||||
list(value).map do |token|
|
||||
field, grain, extra = token.split(':', -1)
|
||||
raise ArgumentError if extra || !field.match?(/\A[a-z_]+\z/) || grain == ''
|
||||
|
||||
grain ? { 'field' => field, 'grain' => grain } : { 'field' => field }
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,7 @@
|
||||
Rails.application.config.to_prepare do
|
||||
[ActiveStorage::Blobs::ProxyController, ActiveStorage::Blobs::RedirectController,
|
||||
ActiveStorage::Representations::ProxyController,
|
||||
ActiveStorage::Representations::RedirectController, ActiveStorage::DiskController].each do |controller|
|
||||
controller.include(HistoryStorageAccess) unless controller < HistoryStorageAccess
|
||||
end
|
||||
end
|
||||
@@ -1,4 +1,19 @@
|
||||
Rails.application.routes.draw do
|
||||
get 'counts/definitions/:resource', to: 'counts_tables#definition'
|
||||
get 'counts/definitions/:resource/candidates', to: 'counts_tables#candidates'
|
||||
get 'counts/query', to: 'counts_tables#index'
|
||||
post 'counts/export/jobs', to: 'counts_exports#create_job'
|
||||
post 'counts/export/jobs/:token/download', to: 'counts_exports#start_download'
|
||||
get 'counts/export/jobs/:token', to: 'counts_exports#job_status'
|
||||
delete 'counts/export/jobs/:token', to: 'counts_exports#cancel_job'
|
||||
post 'counts/export/prepare', to: 'counts_exports#prepare'
|
||||
get 'counts/export/files/:token', to: 'counts_exports#download'
|
||||
get 'counts/export', to: 'counts_exports#index'
|
||||
post 'counts/export', to: 'counts_exports#index'
|
||||
get 'counts/state', to: 'counts_states#index'
|
||||
get 'counts/posts', to: 'counts_posts#index'
|
||||
get 'counts/tags', to: 'counts_tags#index'
|
||||
|
||||
resources :locales, only: [:index]
|
||||
resources :sitemap_entries, only: [:index, :show], param: :resource
|
||||
|
||||
|
||||
@@ -0,0 +1,15 @@
|
||||
namespace :counts do
|
||||
desc 'Process the persistent Counts export queue (run under a process supervisor)'
|
||||
task export_worker: :environment do
|
||||
stopping = false
|
||||
['TERM', 'INT'].each { |signal| Signal.trap(signal) { stopping = true } }
|
||||
at_exit do
|
||||
path = CountsExportJobs.directory.join("#{ CountsExportJobs.worker_id }.json")
|
||||
File.unlink(path) if path.exist?
|
||||
end
|
||||
until stopping
|
||||
worked = Rails.application.executor.wrap { CountsExportJobs.work_one }
|
||||
sleep(1) unless worked
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,69 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe HistoryAccess do
|
||||
include_context 'Ainu locale'
|
||||
|
||||
let!(:tag) { create(:tag) }
|
||||
|
||||
it 'uses the complete tag-description identity and version order at equal timestamps' do
|
||||
locale
|
||||
rows = ['ja', 'ain'].flat_map do |language|
|
||||
['create', 'discard', 'restore'].each_with_index.map do |event, index|
|
||||
TagDescriptionVersion.create!(tag_id: tag.id, language_code: language,
|
||||
version_no: index + 1, event_type: language == 'ain' ? 'update' : event,
|
||||
body: "#{ language } #{ index }", created_at: Time.utc(2024))
|
||||
end
|
||||
end
|
||||
described_class.set(user: nil) do
|
||||
expect(described_class.visible(TagDescriptionVersion.where(id: rows.map(&:id))).pluck(:id))
|
||||
.to match_array([rows[2], *rows.last(3)].map(&:id))
|
||||
expect { described_class.require!(rows.first) }.to raise_error(HistoryAccess::Forbidden)
|
||||
end
|
||||
described_class.set(user: create(:user, :admin)) do
|
||||
expect(described_class.visible(TagDescriptionVersion.where(id: rows.map(&:id))).count).to eq(6)
|
||||
end
|
||||
end
|
||||
|
||||
it 'invalidates public artifacts after a new discard and binds admin artifacts to their actor' do
|
||||
public_metadata = described_class.set(user: nil) { described_class.authority }
|
||||
admin = create(:user, :admin)
|
||||
admin_metadata = described_class.set(user: admin) { described_class.authority }
|
||||
TagVersion.create!(tag:, version_no: 1, event_type: 'discard', name: 'private',
|
||||
category: 'general',
|
||||
names: { ja: { name: 'private', script_code: 'Jpan',
|
||||
auto_generated: false, aliases: [] } },
|
||||
aliases: '', parent_tag_ids: '', created_at: Time.utc(2024))
|
||||
described_class.set(user: nil) do
|
||||
expect(described_class.authorised_artifact?(public_metadata)).to be(false)
|
||||
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
|
||||
end
|
||||
described_class.set(user: create(:user, :admin)) do
|
||||
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
|
||||
end
|
||||
described_class.set(user: admin) do
|
||||
expect(described_class.authorised_artifact?(admin_metadata)).to be(true)
|
||||
admin.update!(role: :guest)
|
||||
expect(described_class.authorised_artifact?(admin_metadata)).to be(false)
|
||||
end
|
||||
end
|
||||
|
||||
it 'denies a blob attached to a discarded post including its derived attachment' do
|
||||
post = create(:post)
|
||||
blob = ActiveStorage::Blob.create_before_direct_upload!(filename: 'private.png',
|
||||
byte_size: 1, checksum: Base64.strict_encode64('checksum'), content_type: 'image/png')
|
||||
ActiveStorage::Attachment.create!(name: 'thumbnail', record: post, blob:)
|
||||
PostVersion.create!(post:, version_no: 1, event_type: 'discard', url: post.url,
|
||||
title: 'private', tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024))
|
||||
variant = ActiveStorage::VariantRecord.create!(blob:, variation_digest: 'spec-variation')
|
||||
derived = ActiveStorage::Blob.create_before_direct_upload!(filename: 'derived.png',
|
||||
byte_size: 1, checksum: Base64.strict_encode64('checksum'), content_type: 'image/png')
|
||||
ActiveStorage::Attachment.create!(name: 'image', record: variant, blob: derived)
|
||||
described_class.set(user: nil) do
|
||||
expect(described_class.blob_visible?(blob)).to be(false)
|
||||
expect(described_class.blob_visible?(derived)).to be(false)
|
||||
end
|
||||
described_class.set(user: create(:user, :admin)) do
|
||||
expect(described_class.blob_visible?(derived)).to be(true)
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,40 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe 'History boundaries shared with Counts', type: :request do
|
||||
let!(:tag) { create(:tag, primary_name: 'description tag') }
|
||||
let!(:description) do
|
||||
TagDescription.create!(tag:, language_code: 'ja', version_no: 2, body: 'private body')
|
||||
end
|
||||
|
||||
before do
|
||||
['create', 'discard'].each_with_index do |event, index|
|
||||
TagDescriptionVersion.create!(tag_id: tag.id, language_code: 'ja',
|
||||
version_no: index + 1, event_type: event, body: 'private body',
|
||||
created_at: Time.utc(2024))
|
||||
end
|
||||
end
|
||||
|
||||
it 'denies the same discarded description through Counts, the facade and the ordinary API' do
|
||||
get '/counts/query', params: { resource: 'tag_descriptions', definition: 'history',
|
||||
mode: 'details', locale: 'ja', execution_at: '2025-01-01T00:00:00Z' }
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(response.parsed_body).to include('count' => 0, 'output_count' => 2,
|
||||
'excluded_count' => 2, 'versions' => [])
|
||||
["/wiki/#{ description.id }", "/tags/#{ tag.id }/description"].each do |path|
|
||||
get path, params: { locale: 'ja' }
|
||||
expect(response).to have_http_status(:not_found)
|
||||
expect(response.body).not_to include('private body')
|
||||
end
|
||||
get '/sitemap_entries/descriptions', params: { locale: 'ja' }
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(response.parsed_body['ids']).not_to include(description.id)
|
||||
end
|
||||
|
||||
it 'permits the server-authenticated administrator to read the discarded description' do
|
||||
sign_in_as(create(:user, :admin))
|
||||
get "/tags/#{ tag.id }/description", params: { locale: 'ja' }
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(response.parsed_body['body']).to eq('private body')
|
||||
expect(response.headers['Cache-Control']).to include('no-store')
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,97 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe 'Legacy Counts API', type: :request do
|
||||
let!(:actor) { create(:user, :member) }
|
||||
let!(:post) { create(:post) }
|
||||
let!(:tag) { create(:tag, primary_name: 'counts tag', category: :general) }
|
||||
|
||||
before do
|
||||
snapshot = { tag_id: tag.id, version_no: 1, name: 'counts tag',
|
||||
category: 'general', sections: [] }
|
||||
[1, 2, 3].each do |number|
|
||||
time = Time.utc(2024, number == 3 ? 2 : 1, number, 0, 0)
|
||||
PostVersion.create!(post:, version_no: number,
|
||||
event_type: number == 1 ? 'create' : 'update',
|
||||
tags: '', url: post.url, title: "Snapshot #{ number }", parent_post_ids: '',
|
||||
tags_json: number == 1 ? [snapshot] : [],
|
||||
created_by_user: number == 3 ? nil : actor, created_at: time)
|
||||
TagVersion.create!(tag:, version_no: number,
|
||||
event_type: number == 1 ? 'create' : 'update',
|
||||
name: 'counts tag', category: 'general', aliases: '', parent_tag_ids: '',
|
||||
names: { ja: { name: 'counts tag', script_code: 'Jpan',
|
||||
auto_generated: false, aliases: [] } },
|
||||
created_by_user: number == 3 ? nil : actor, created_at: time)
|
||||
end
|
||||
end
|
||||
|
||||
['posts', 'tags'].each do |resource|
|
||||
context resource do
|
||||
let(:path) { "/counts/#{ resource }" }
|
||||
let(:id) { resource == 'posts' ? post.id : tag.id }
|
||||
let(:filters) { { resource_id: id, user_id: actor.id, latest: '0',
|
||||
from: '2024-01-01T00:00Z', before: '2024-02-01T00:00Z' } }
|
||||
|
||||
it 'uses the same AND filters for SQL totals, groups and details' do
|
||||
get path, params: filters.merge(axis: 'month')
|
||||
expect(response).to have_http_status(:ok)
|
||||
summary = response.parsed_body
|
||||
expect(summary['count']).to eq(2)
|
||||
expect(summary['groups'].sum { _1['count'] }).to eq(2)
|
||||
expect(summary['groups'].map { _1['key'] }).to eq(['2024-01'])
|
||||
|
||||
get path, params: filters.merge(mode: 'details')
|
||||
expect(response.parsed_body['count']).to eq(2)
|
||||
expect(response.parsed_body['versions'].map { _1['version_no'] }).to eq([2, 1])
|
||||
end
|
||||
|
||||
it 'paginates details and groups independently from totals' do
|
||||
get path, params: { resource_id: id, latest: '0', mode: 'details', limit: 1, page: 2 }
|
||||
expect(response.parsed_body['count']).to eq(3)
|
||||
expect(response.parsed_body['versions'].sole['version_no']).to eq(2)
|
||||
get path, params: { resource_id: id, latest: '0', axis: 'month', limit: 1, page: 2 }
|
||||
expect(response.parsed_body['group_count']).to eq(2)
|
||||
expect(response.parsed_body['groups'].sole).to include('key' => '2024-02', 'count' => 1)
|
||||
end
|
||||
|
||||
it 'groups null actors and supports their drill-down filter' do
|
||||
get path, params: { resource_id: id, latest: '0', axis: 'user' }
|
||||
expect(response.parsed_body['groups']).to include(
|
||||
{ 'key' => nil, 'name' => nil, 'count' => 1 })
|
||||
get path, params: { resource_id: id, mode: 'details', user_id: 'none' }
|
||||
expect(response.parsed_body['count']).to eq(1)
|
||||
end
|
||||
|
||||
it 'uses an inclusive start and exclusive end' do
|
||||
get path, params: { resource_id: id, latest: '0',
|
||||
from: '2024-01-01T00:00Z', before: '2024-01-02T00:00Z' }
|
||||
expect(response.parsed_body['count']).to eq(1)
|
||||
end
|
||||
|
||||
it 'rejects unsupported or malformed query values' do
|
||||
[
|
||||
{ axis: 'SQL' }, { limit: 101 }, { page: '1x' }, { user_id: '-1' },
|
||||
{ event_type: 'invalid' }, { columns: 'password' }, { order: 'id' },
|
||||
{ from: '2024-02-30T00:00Z' }, { from: '2024-01-01' },
|
||||
{ user_id: ['1'] }, { mode: 'invalid' }].each do |invalid|
|
||||
get path, params: invalid
|
||||
expect(response).to have_http_status(:bad_request)
|
||||
end
|
||||
end
|
||||
|
||||
it 'retains the existing banned-user boundary' do
|
||||
actor.update!(banned_at: Time.current)
|
||||
sign_in_as(actor)
|
||||
get path
|
||||
expect(response).to have_http_status(:forbidden)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
it 'matches removed tags from the previous snapshot, not current post_tags' do
|
||||
get '/counts/posts', params: { tag: tag.id, latest: '0', mode: 'details' }
|
||||
expect(response.parsed_body['count']).to eq(2)
|
||||
expect(response.parsed_body['versions'].map { _1['version_no'] }).to eq([2, 1])
|
||||
expect(response.parsed_body['versions'].first['tags']).to be_empty
|
||||
expect(response.parsed_body['versions'].last['tags'].sole['id']).to eq(tag.id)
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,248 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe 'Counts table API', type: :request do
|
||||
include_context 'Ainu locale'
|
||||
|
||||
let!(:actor) { create(:user, :member, name: 'Counts actor') }
|
||||
let!(:record) { create(:post) }
|
||||
let!(:tag) { create(:tag, primary_name: 'alpha', category: :general) }
|
||||
let!(:external) { create(:external_tag, name: 'beta') }
|
||||
let(:point) { '2025-01-01T00:00:00Z' }
|
||||
let(:base) do
|
||||
{ resource: 'posts', definition: 'history', mode: 'details',
|
||||
execution_at: point, timezone: 'UTC', locale: 'ja' }
|
||||
end
|
||||
|
||||
before do
|
||||
TagVersion.create!(tag:, version_no: 1, event_type: 'create', name: 'alpha',
|
||||
category: 'general', aliases: '', parent_tag_ids: '',
|
||||
names: { ja: { name: 'alpha', script_code: 'Jpan', auto_generated: false, aliases: [] } },
|
||||
created_at: Time.utc(2023))
|
||||
NicoTagVersion.create!(external_tag: external, version_no: 1,
|
||||
event_type: 'create', name: 'beta', linked_tags: '', created_at: Time.utc(2023))
|
||||
snapshots = [
|
||||
{ tag_id: tag.id, version_no: 1, name: 'alpha', category: 'general', sections: [] },
|
||||
{ external_tag_id: external.id }]
|
||||
[1, 2, 3].each do |number|
|
||||
PostVersion.create!(post: record, version_no: number,
|
||||
event_type: number == 1 ? 'create' : 'update', url: record.url,
|
||||
title: "snapshot #{ number }", tags: '', parent_post_ids: '',
|
||||
tags_json: snapshots.first(number == 1 ? 2 : (number == 2 ? 1 : 0)),
|
||||
created_by_user: number == 3 ? nil : actor,
|
||||
created_at: Time.utc(2024, 1, 31, 15) + (number - 1).hours)
|
||||
end
|
||||
end
|
||||
|
||||
def query(extra = {})
|
||||
get '/counts/query', params: base.merge(extra)
|
||||
expect(response).to have_http_status(:ok)
|
||||
response.parsed_body
|
||||
end
|
||||
|
||||
def axis(field, value, type: 'string', grain: nil, root: 'conditions')
|
||||
result = { root => '', "#{ root }.0.field" => field,
|
||||
"#{ root }.0.value_type" => type }
|
||||
result["#{ root }.0.operator"] = 'axis_eq' if root == 'conditions'
|
||||
result["#{ root }.0.value"] = value unless type == 'null'
|
||||
result["#{ root }.0.grain"] = grain if grain
|
||||
result
|
||||
end
|
||||
|
||||
def versions(result) = result.fetch('versions').map { _1.fetch('version_no') }
|
||||
|
||||
it 'separates latest and history and defaults to stable detail ordering and limit 100' do
|
||||
history = query
|
||||
expect(versions(history)).to eq([1, 2, 3])
|
||||
expect(history).to include('limit' => 100, 'count' => 3, 'output_count' => 3,
|
||||
'visible_output_count' => 3, 'excluded_count' => 0)
|
||||
expect(history['versions'].map { _1['latest'] }).to eq([false, false, true])
|
||||
expect(versions(query(definition: 'latest'))).to eq([3])
|
||||
expect(versions(query(limit: 1, page: 2))).to eq([2])
|
||||
expect(versions(query(sort: 'version_no:desc', limit: 1))).to eq([3])
|
||||
end
|
||||
|
||||
it 'counts regular pivot rows separately from totals and from underlying records' do
|
||||
total = query(mode: 'summary', rows: '', cols: '')
|
||||
expect(total).to include('count' => 3, 'output_count' => 1, 'group_count' => 1)
|
||||
expect(total['cells']).to eq([{ 'values' => [], 'count' => 3 }])
|
||||
pivot = query(mode: 'summary', rows: 'event_type', cols: '', sort: '', limit: 1)
|
||||
expect(pivot).to include('group_count' => 2, 'visible_output_count' => 2)
|
||||
expect(pivot['row_headers'].count { _1['kind'] == 'data' }).to eq(1)
|
||||
expect(pivot['row_headers']).to include(hash_including('kind' => 'grand_total'))
|
||||
expect(query(mode: 'summary', rows: '', cols: 'event_type', sort: '')['output_count']).to eq(1)
|
||||
end
|
||||
|
||||
it 'preserves timezone, NULL and legacy drill semantics and permits condition removal' do
|
||||
month = axis('created_at', '2024-02', grain: 'month')
|
||||
expect(versions(query(month.merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
|
||||
expect(versions(query(month))).to eq([])
|
||||
old = axis('created_at', '2024-02', grain: 'month', root: 'drill')
|
||||
expect(versions(query(old.merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
|
||||
expect(versions(query(old.merge(month).merge(timezone: 'Asia/Tokyo')))).to eq([1, 2, 3])
|
||||
expect(versions(query(axis('user', nil, type: 'null')))).to eq([3])
|
||||
expect(versions(query(axis('changed_field', 'title')))).to eq([2, 3])
|
||||
expect(versions(query)).to eq([1, 2, 3])
|
||||
end
|
||||
|
||||
it 'evaluates latest before filtering the version set' do
|
||||
predicate = { conditions: '', 'conditions.0.field' => 'latest',
|
||||
'conditions.0.operator' => 'eq', 'conditions.0.value' => 'false' }
|
||||
expect(versions(query(predicate))).to eq([1, 2])
|
||||
expect(versions(query(predicate.merge('conditions.0.value' => 'true')))).to eq([3])
|
||||
end
|
||||
|
||||
it 'keeps ANY, ALL and NONE distinct without multiplying parent records' do
|
||||
values = { conditions: '', 'conditions.0.field' => 'tags',
|
||||
'conditions.0.values' => '1', condition_values: '',
|
||||
'condition_values.0.condition' => '0',
|
||||
'condition_values.0.value_type' => 'string',
|
||||
'condition_values.0.value' => "internal:#{ tag.id }",
|
||||
'condition_values.1.condition' => '0',
|
||||
'condition_values.1.value_type' => 'string',
|
||||
'condition_values.1.value' => "external:#{ external.id }" }
|
||||
{ 'any' => [1, 2], 'all' => [1], 'none' => [3] }.each do |operator, expected|
|
||||
expect(versions(query(values.merge('conditions.0.operator' => operator)))).to eq(expected)
|
||||
end
|
||||
result = query(axis('tags', "internal:#{ tag.id }"))
|
||||
expect(result['versions'].first['tags'].length).to eq(2)
|
||||
end
|
||||
|
||||
it 'distinguishes IN, NOT IN, an empty selection and an unspecified selection' do
|
||||
predicate = { conditions: '', 'conditions.0.field' => 'version_no',
|
||||
'conditions.0.operator' => 'in', 'conditions.0.values' => '1',
|
||||
condition_values: '', 'condition_values.0.condition' => '0',
|
||||
'condition_values.0.value_type' => 'number',
|
||||
'condition_values.0.value' => '1' }
|
||||
expect(versions(query(predicate))).to eq([1])
|
||||
expect(versions(query(predicate.merge('conditions.0.operator' => 'not_in')))).to eq([2, 3])
|
||||
empty = predicate.reject { |key, _value| key.to_s.start_with?('condition_values.') }
|
||||
expect(versions(query(empty))).to be_empty
|
||||
expect(versions(query(empty.except('conditions.0.values')))).to eq([1, 2, 3])
|
||||
end
|
||||
|
||||
it 'canonicalises unreferenced tags while preserving partial stable references' do
|
||||
reference = { tag_refs: '', 'tag_refs.0.name' => 'alpha',
|
||||
'tag_refs.0.tag_id' => tag.id.to_s }
|
||||
expect(versions(query(tags: 'alpha'))).to eq([1, 2])
|
||||
expect(versions(query(reference.merge(tags: 'alpha nico:beta')))).to eq([1])
|
||||
expect(versions(query(reference.merge(tags: 'nico:beta alpha')))).to eq([1])
|
||||
expect(versions(query(reference.merge(tags: 'alpha not:nico:beta')))).to eq([2])
|
||||
expect(versions(query(reference.merge(tags: 'alpha nico:beta', match: 'any')))).to eq([1, 2])
|
||||
expect(versions(query(tags: 'not:alpha'))).to eq([3])
|
||||
get '/counts/query', params: base.merge(reference).merge(tags: 'different')
|
||||
expect(response).to have_http_status(:bad_request)
|
||||
end
|
||||
|
||||
it 'rejects malformed structured conditions and axes without accepting arbitrary SQL' do
|
||||
[
|
||||
{ rows: '[{"field":"user"}]' }, { rows: 'user user' },
|
||||
{ rows: 'user:month' }, { rows: 'password' }, { limit: 101 }, { limit: 0 },
|
||||
{ page: '1x' }, { conditions: 'eyJmaWVsZCI6InVzZXIifQ==' },
|
||||
{ conditions: '', 'conditions.1.field' => 'user' },
|
||||
axis('password', '1'), axis('changed_field', 'title').merge(definition: 'latest'),
|
||||
axis('user', 'yes', type: 'boolean'),
|
||||
{ conditions: '', 'conditions.0.field' => 'title',
|
||||
'conditions.0.operator' => 'eq', 'conditions.0.value' => 'x' * 8193 }].each do |invalid|
|
||||
get '/counts/query', params: base.merge(invalid)
|
||||
expect(response).to have_http_status(:bad_request), invalid.inspect
|
||||
end
|
||||
end
|
||||
|
||||
it 'rejects duplicate URL keys and JSON query payloads at the HTTP boundary' do
|
||||
['resource=posts&limit=20&limit=50', 'resource=posts&headers=%5B%22Id.%22%5D',
|
||||
'resource=posts&limit%5B%5D=20'].each do |query_string|
|
||||
get "/counts/query?#{ query_string }"
|
||||
expect(response).to have_http_status(:bad_request)
|
||||
end
|
||||
end
|
||||
|
||||
it 'reports column overflow separately from malformed input' do
|
||||
101.times do |index|
|
||||
PostVersion.create!(post: record, version_no: index + 4,
|
||||
event_type: 'update', url: record.url, title: "column #{ index }",
|
||||
tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 3))
|
||||
end
|
||||
get '/counts/query', params: base.merge(mode: 'summary', rows: '', cols: 'title', sort: '')
|
||||
expect(response).to have_http_status(:unprocessable_entity)
|
||||
expect(response.parsed_body).to eq('error' => 'too_many_columns', 'max_columns' => 100)
|
||||
end
|
||||
|
||||
it 'defines all six resources with filterable output fields and shared detail defaults' do
|
||||
locale
|
||||
CountsResourceDefinition::DEFINITIONS.each_key do |resource|
|
||||
get "/counts/definitions/#{ resource }", params: { locale: 'ain-Kana' }
|
||||
expect(response).to have_http_status(:ok)
|
||||
body = response.parsed_body
|
||||
filters = body['filter_fields'].map { _1['key'] }
|
||||
walk = lambda do |fields, prefix|
|
||||
fields.each do |field|
|
||||
key = [prefix, field['key']].compact.join('.')
|
||||
expect(filters).to include(key)
|
||||
walk.call(field.fetch('children', []), key)
|
||||
end
|
||||
end
|
||||
walk.call(body['output_fields'], nil)
|
||||
defaults = body['defaults'].select { _1['mode'] == 'details' }
|
||||
expect(defaults.map { _1['definition'] }).to eq(['latest', 'history'])
|
||||
expect(defaults.first['params']).to eq(defaults.last['params'])
|
||||
settings = defaults.first['params'].to_h { [_1['key'], _1['value']] }
|
||||
expect(settings['sort']).to eq('resource_id:asc version_no:asc')
|
||||
if resource == 'posts'
|
||||
expect(settings['children'].split).to eq(['tags.name_ain', 'parents.id'])
|
||||
end
|
||||
keys = body['name_columns'].map { _1['key'] }
|
||||
languages = Language.order(:created_at, :code).pluck(:code)
|
||||
expect(keys).to eq(languages.map { "name_#{ _1 }" } + languages.map { "aliases_#{ _1 }" })
|
||||
end
|
||||
end
|
||||
|
||||
[nil, :guest, :member, :admin].each do |role|
|
||||
context "authority #{ role || 'anonymous' }" do
|
||||
before do
|
||||
sign_in_as(create(:user, role)) if role
|
||||
# Equal timestamps deliberately require version ordering, not a date comparison.
|
||||
['discard', 'restore', 'discard', 'restore'].each_with_index do |event, index|
|
||||
PostVersion.create!(post: record, version_no: index + 4, event_type: event,
|
||||
url: record.url, title: event == 'restore' ? 'restored' : 'private snapshot',
|
||||
tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024, 3))
|
||||
end
|
||||
end
|
||||
|
||||
it 'keeps every earlier discard boundary private but counts denied records' do
|
||||
result = query
|
||||
expected = role == :admin ? (1..7).to_a : [7]
|
||||
expect(versions(result)).to eq(expected)
|
||||
expect(result).to include('output_count' => 7,
|
||||
'visible_output_count' => expected.length, 'excluded_count' => 7 - expected.length)
|
||||
expect(response.headers['Cache-Control']).to include('no-store')
|
||||
expect(response.body).not_to include('private snapshot') unless role == :admin
|
||||
end
|
||||
|
||||
it 'applies the same version boundary to the ordinary history endpoint and old values' do
|
||||
get '/posts/versions', params: { post: record.id, locale: 'ja' }
|
||||
expect(response).to have_http_status(:ok)
|
||||
rows = response.parsed_body.fetch('versions')
|
||||
expect(rows.map { _1['version_no'] }).to eq(role == :admin ? (1..7).to_a.reverse : [7])
|
||||
unless role == :admin
|
||||
expect(rows.sole.dig('title', 'prev')).to be_nil
|
||||
expect(response.body).not_to include('private snapshot')
|
||||
end
|
||||
end
|
||||
|
||||
it 'does not resolve private condition names even when the result set is empty' do
|
||||
TagVersion.create!(tag:, version_no: 2, event_type: 'discard', name: 'secret tag',
|
||||
category: 'general', aliases: '', parent_tag_ids: '',
|
||||
names: { ja: { name: 'secret tag', script_code: 'Jpan', aliases: [] } },
|
||||
created_at: Time.utc(2024, 2))
|
||||
result = query(axis('tags', "internal:#{ tag.id }").merge(definition: 'latest'))
|
||||
expect(result['versions']).to be_empty
|
||||
expected = role == :admin ? { "internal:#{ tag.id }" => 'secret tag' } : {}
|
||||
expect(result['condition_labels']['tags']).to eq(expected)
|
||||
get '/counts/definitions/posts/candidates', params: {
|
||||
field: 'tags', query: 'secret', execution_at: point, locale: 'ja' }
|
||||
expect(response).to have_http_status(:ok)
|
||||
expect(response.parsed_body['rows']).to be_empty
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -21,7 +21,29 @@ RSpec.describe 'Post filter identity across locales', type: :request do
|
||||
|
||||
['nico:shared', 'union_alias', 'not:nico:shared'].each do |literal|
|
||||
it "preserves the complete match set for #{ literal }" do
|
||||
get '/posts', params: { locale: 'ja', tags: literal }
|
||||
reference_params = { tag_refs: '' }
|
||||
|
||||
references.each_with_index do |reference, index|
|
||||
%w[name tag_id external_tag_id negative missing].each do |field|
|
||||
value = reference[field]
|
||||
next if value.nil?
|
||||
next if field == 'missing' && value == false
|
||||
|
||||
reference_params["tag_refs.#{index}.#{field}"] =
|
||||
case value
|
||||
when true
|
||||
'1'
|
||||
when false
|
||||
'0'
|
||||
else
|
||||
value.to_s
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
get '/posts', params: {
|
||||
locale: 'ain-Kana',
|
||||
tags: references.map { _1.fetch('name') }.join(' ') }.merge(reference_params)
|
||||
expect(response).to have_http_status(:ok)
|
||||
original_ids = json.fetch('posts').map { _1.fetch('id') }
|
||||
expected = literal.start_with?('not:') ? [other_post.id] :
|
||||
|
||||
@@ -55,7 +55,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
|
||||
end
|
||||
|
||||
it 'does not fall back to a revision absent from new history' do
|
||||
versions.first.destroy!
|
||||
TagDescriptionVersion.where(id: versions.first.id).delete_all
|
||||
without_legacy_reads do
|
||||
get "/wiki/#{ description.id }", params: { version: versions.first.id }
|
||||
end
|
||||
@@ -190,7 +190,7 @@ RSpec.describe 'Tag description Wiki compatibility', type: :request do
|
||||
describe 'index' do
|
||||
[nil, 'description_contract'].each do |title|
|
||||
it "returns new content once in #{ title ? 'filtered' : 'unfiltered' } index" do
|
||||
get '/wiki', params: { title: }
|
||||
get '/wiki', params: title.nil? ? { } : { title: title }
|
||||
expect(response).to have_http_status(:ok)
|
||||
rows = json.select { |row| row.fetch('id') == description.id }
|
||||
expect(rows.size).to eq(1)
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
require 'rails_helper'
|
||||
require 'tmpdir'
|
||||
require 'csv'
|
||||
|
||||
RSpec.describe 'Counts export contracts' do
|
||||
include ActiveSupport::Testing::TimeHelpers
|
||||
let(:payload) do
|
||||
{ 'resource' => 'posts', 'definition' => 'history', 'mode' => 'details',
|
||||
'locale' => 'ja', 'execution_at' => '2024-03-01T00:00:00Z',
|
||||
'columns' => 'title', 'headers' => '["Title"]', 'format' => 'json' }
|
||||
end
|
||||
|
||||
around do |example|
|
||||
Dir.mktmpdir('counts-export-spec') do |path|
|
||||
@counts_export_dir = Pathname.new(path)
|
||||
HistoryAccess.set(user: nil) { example.run }
|
||||
end
|
||||
end
|
||||
|
||||
before do
|
||||
allow(CountsExportDownload).to receive(:directory)
|
||||
.and_return(@counts_export_dir)
|
||||
end
|
||||
|
||||
context 'serialization after the authorised query' do
|
||||
let(:rows) { [{ title: "=formula\n\"quoted\"" }, { title: 'second' }] }
|
||||
let(:page) do
|
||||
{ count: 2, output_count: 2, visible_output_count: 2, page: 1, limit: 100,
|
||||
versions: rows, execution_at: payload.fetch('execution_at') }
|
||||
end
|
||||
|
||||
before do
|
||||
query = CountsTableQuery.new(payload.slice(*CountsTableQuery::KEYS))
|
||||
# SQL/authority are exercised in request specs; this unit boundary supplies their page.
|
||||
allow(query).to receive(:each_page) do |before_page:, &block|
|
||||
before_page.call
|
||||
block.call(page)
|
||||
end
|
||||
allow(CountsTableQuery).to receive(:new).and_return(query)
|
||||
allow(ApplicationRecord).to receive(:transaction).with(isolation: :repeatable_read).and_yield
|
||||
end
|
||||
|
||||
it 'reports aggregation, actual output progress and regular-row metadata' do
|
||||
progress = []
|
||||
phases = []
|
||||
body = CountsTableExport.new(payload).body(
|
||||
progress: ->(done, total) { progress << [done, total] },
|
||||
phase: ->(value) { phases << value }).to_a.join
|
||||
result = JSON.parse(body)
|
||||
expect(result['data']).to eq(rows.map(&:stringify_keys))
|
||||
expect(result['metadata']).to include('logical_count_unit' => 'regular_result_rows',
|
||||
'aggregate_records_in_logical_count' => false, 'output_count' => 2)
|
||||
expect(phases).to eq(['aggregating', 'writing'])
|
||||
expect(progress).to include([0, 2], [1, 2], [2, 2])
|
||||
expect(progress.last).to eq([2, 2])
|
||||
end
|
||||
|
||||
it 'preserves CSV multiline text and quotes while neutralising spreadsheet formulas' do
|
||||
body = CountsTableExport.new(payload.merge('format' => 'csv')).body.to_a.join
|
||||
expect(CSV.parse(body.delete_prefix("\uFEFF")))
|
||||
.to eq([['Title'], ["'=formula\n\"quoted\""], ['second']])
|
||||
end
|
||||
|
||||
it 'completes an isolated worker job with its real processed count and cancellable file' do
|
||||
CountsExportJobs.pulse
|
||||
token = CountsExportJobs.start(payload, 'owner', 'ja')
|
||||
id = CountsExportJobs.authorised_id(token, 'owner')
|
||||
state = CountsExportJobs.read(id).merge('status' => 'running', 'phase' => 'aggregating')
|
||||
CountsExportJobs.write(state)
|
||||
# Invoke the worker entrypoint directly; never fork or start an external worker in specs.
|
||||
CountsExportJobs.perform(state)
|
||||
ready = CountsExportJobs.status(token, 'owner')
|
||||
expect(ready).to include('status' => 'ready', 'processed' => 2, 'total' => 2)
|
||||
file = CountsExportDownload.find(ready.fetch('token'), 'owner')
|
||||
expect(JSON.parse(File.read(file.fetch('path')))['data'].length).to eq(2)
|
||||
CountsExportJobs.status(token, 'owner', cancel: true)
|
||||
expect(CountsExportDownload.find(ready.fetch('token'), 'owner')).to be_nil
|
||||
end
|
||||
|
||||
it 'marks a generation exception as failed without issuing a download token' do
|
||||
CountsExportJobs.pulse
|
||||
token = CountsExportJobs.start(payload, 'owner', 'ja')
|
||||
id = CountsExportJobs.authorised_id(token, 'owner')
|
||||
state = CountsExportJobs.read(id).merge('status' => 'running')
|
||||
CountsExportJobs.write(state)
|
||||
allow(CountsExportDownload).to receive(:prepare).and_raise(CountsExportDownload::TooLarge)
|
||||
CountsExportJobs.perform(state)
|
||||
expect(CountsExportJobs.status(token, 'owner')).to include('status' => 'failed')
|
||||
expect(CountsExportJobs.status(token, 'owner')).not_to have_key('token')
|
||||
end
|
||||
end
|
||||
|
||||
it 'retains multilingual names and all inner rows in JSON with one selected display column' do
|
||||
names = [{ language_code: 'ja', name: 'alpha', script_code: 'Jpan', aliases: [] },
|
||||
{ language_code: 'ain', name: 'recorded', script_code: 'Kana',
|
||||
aliases: [{ name: 'alias', script_code: 'Latn' }] }]
|
||||
rows = [{ id: 1, name_ja: 'alpha', names: }, { id: 2, name_ja: 'beta', names: }]
|
||||
projection = CountsExportProjection.new(payload.merge('columns' => 'tags',
|
||||
'headers' => '["Tags"]', 'children' => 'tags.name_ja'),
|
||||
resource: 'posts', summary: false, tags: false)
|
||||
result = projection.project({ tags: rows })
|
||||
expect(result['tags'].length).to eq(2)
|
||||
expect(result['tags'].map { _1['name_ja'] }).to eq(['alpha', 'beta'])
|
||||
expect(result['tags'].map { _1['names'] }).to eq([names, names])
|
||||
end
|
||||
|
||||
it 'requires a live worker, enforces queue capacity and binds jobs to their owner' do
|
||||
expect { CountsExportJobs.start(payload, 'owner', 'ja') }
|
||||
.to raise_error(CountsExportJobs::WorkerUnavailable)
|
||||
CountsExportJobs.pulse
|
||||
token = CountsExportJobs.start(payload, 'owner', 'ja')
|
||||
expect(CountsExportJobs.status(token, 'other')).to be_nil
|
||||
expect(CountsExportJobs.status(token, 'owner')).to include(
|
||||
'status' => 'queued', 'phase' => 'queued', 'processed' => 0, 'total' => nil)
|
||||
(CountsExportDownload::MAX_CONCURRENT_EXPORTS - 1).times do
|
||||
CountsExportJobs.start(payload, 'owner', 'ja')
|
||||
end
|
||||
expect { CountsExportJobs.start(payload, 'owner', 'ja') }
|
||||
.to raise_error(CountsExportDownload::Busy)
|
||||
expect(CountsExportJobs.status(token, 'owner', cancel: true)['status']).to eq('cancelled')
|
||||
end
|
||||
|
||||
it 'fails stale queued/running jobs and preserves actual progress without a fake total' do
|
||||
CountsExportJobs.pulse
|
||||
token = CountsExportJobs.start(payload, 'owner', 'ja')
|
||||
id = CountsExportJobs.authorised_id(token, 'owner')
|
||||
state = CountsExportJobs.read(id)
|
||||
state.merge!('status' => 'running', 'phase' => 'aggregating')
|
||||
CountsExportJobs.write(state)
|
||||
CountsExportJobs.checkpoint(id, 5, nil, phase: 'writing')
|
||||
expect(CountsExportJobs.status(token, 'owner')).to include(
|
||||
'phase' => 'writing', 'processed' => 5, 'total' => nil)
|
||||
state = CountsExportJobs.read(id)
|
||||
state['heartbeat_at'] = (Time.current - CountsExportJobs::HEARTBEAT_SECONDS - 1).to_f
|
||||
CountsExportJobs.write(state)
|
||||
expect(CountsExportJobs.status(token, 'owner')).to include(
|
||||
'status' => 'failed', 'error' => 'worker_stalled', 'processed' => 5)
|
||||
expect { CountsExportJobs.checkpoint(id, 6) }.to raise_error(CountsExportJobs::Cancelled)
|
||||
end
|
||||
|
||||
it 'revokes an already generated download on a new discard and rejects wrong owners' do
|
||||
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['{}'])
|
||||
token = CountsExportDownload.prepare(exporter, 'owner')
|
||||
expect(CountsExportDownload.find(token, 'owner')).not_to be_nil
|
||||
expect(CountsExportDownload.find(token, 'other')).to be_nil
|
||||
post = create(:post)
|
||||
PostVersion.create!(post:, version_no: 1, event_type: 'discard', url: post.url,
|
||||
title: 'private', tags: '', tags_json: [], parent_post_ids: '', created_at: Time.utc(2024))
|
||||
expect(CountsExportDownload.find(token, 'owner')).to be_nil
|
||||
end
|
||||
|
||||
it 'rejects oversized output and removes its partial file' do
|
||||
stub_const('CountsExportDownload::MAX_FILE_BYTES', 4)
|
||||
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['12345'])
|
||||
expect { CountsExportDownload.prepare(exporter, 'owner') }
|
||||
.to raise_error(CountsExportDownload::TooLarge)
|
||||
expect(Dir.glob(CountsExportDownload.directory.join('counts-*'))).to be_empty
|
||||
end
|
||||
|
||||
it 'expires queued work after worker loss and rejects expired download signatures' do
|
||||
CountsExportJobs.pulse
|
||||
token = CountsExportJobs.start(payload, 'owner', 'ja')
|
||||
travel_to(Time.current + CountsExportJobs::HEARTBEAT_SECONDS + 1) do
|
||||
expect(CountsExportJobs.status(token, 'owner')).to include(
|
||||
'status' => 'failed', 'error' => 'worker_unavailable')
|
||||
end
|
||||
exporter = instance_double(CountsExporter, resource: 'posts', format: 'json', body: ['{}'])
|
||||
download = CountsExportDownload.prepare(exporter, 'owner')
|
||||
travel_to(Time.current + CountsExportDownload::EXPIRY + 1) do
|
||||
expect(CountsExportDownload.find(download, 'owner')).to be_nil
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,33 @@
|
||||
require 'rails_helper'
|
||||
|
||||
RSpec.describe QueryState do
|
||||
it 'decodes typed axis values and keeps absence distinct from an empty set' do
|
||||
{ 'boolean' => ['false', false], 'number' => ['7', 7],
|
||||
'string' => ['a b & c', 'a b & c'], 'null' => ['', nil] }.each do |type, (raw, expected)|
|
||||
params = { 'conditions' => '', 'conditions.0.field' => 'user',
|
||||
'conditions.0.operator' => 'axis_eq',
|
||||
'conditions.0.value_type' => type, 'conditions.0.value' => raw }
|
||||
expect(described_class.records(params, 'conditions').sole['value']).to eq(expected)
|
||||
end
|
||||
expect(described_class.records({}, 'conditions')).to eq([])
|
||||
expect(described_class.records({ 'conditions' => '' }, 'conditions')).to eq([])
|
||||
expect(described_class.records({ 'conditions' => '', 'conditions.0.values' => '1' },
|
||||
'conditions')).to eq([{ 'values' => true }])
|
||||
end
|
||||
|
||||
it 'rejects gaps, unknown keys, JSON, wrong types and oversized lists' do
|
||||
[
|
||||
{ 'conditions' => '[]' }, { 'conditions' => '', 'conditions.1.field' => 'user' },
|
||||
{ 'conditions' => '', 'conditions.0.sql' => 'TRUE' },
|
||||
{ 'conditions' => '', 'conditions.0.blank' => 'true' },
|
||||
{ 'conditions' => '', 'conditions.0.value' => ['x'] },
|
||||
{ 'drill' => '', 'drill.0.field' => 'user', 'drill.0.value_type' => 'object' },
|
||||
{ 'conditions' => 'x' * 131_073 }].each do |params|
|
||||
expect { described_class.validate!(params, ['conditions', 'drill']) }
|
||||
.to raise_error(ArgumentError)
|
||||
end
|
||||
['user user', 'user title', ' user', (1..201).to_a.join(' ')].each do |value|
|
||||
expect { described_class.list(value) }.to raise_error(ArgumentError)
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,107 @@
|
||||
# Counts エクスポートワーカー
|
||||
|
||||
Counts の画面からのエクスポートは永続ファイルキューへ登録する。
|
||||
Active Job の既定のプロセス内 async アダプターは使用しない。
|
||||
追加 gem、外部キュー、DB マイグレーションは不要。
|
||||
|
||||
## 起動・配置
|
||||
|
||||
backend で `bundle exec rake counts:export_worker` を、systemd 等の監視対象として
|
||||
別プロセスで起動する。Web プロセス内では起動しない。
|
||||
1監視プロセスは同時に1ジョブを専用の子プロセスで生成し、全体の実行枠は
|
||||
既存の2枠を共有する。fork 前に DB 接続を閉じ、子が独立した接続を取得する。
|
||||
Web 内のスレッドや Active Job async では実行しない。Linux / POSIX の fork が必要。
|
||||
監視側は SQL 中も1秒間隔でジョブとワーカーの heartbeat を更新する。
|
||||
|
||||
`docs/systemd/btrc-counts-exports.service` を配備用 unit として追加している。
|
||||
Restart=always、KillMode=control-group、停止猶予150秒を指定している。
|
||||
User、WorkingDirectory、bundle が見つかる PATH を実際の配備先へ合わせる。
|
||||
`/etc/btrc-hub/counts-exports.env` に Web と同じ Rails secret・接続設定・
|
||||
COUNTS_EXPORT_DIRECTORY を設定し、同じ Unix ユーザーで読み書きできるようにする。
|
||||
OS 起動時の有効化・サービス起動は運用側で行う必要がある。
|
||||
今回 unit のインストール、有効化、本番での起動・再起動は行っていない。
|
||||
|
||||
Web とワーカーは同一リリース、Rails secret、アプリケーション環境、DB 接続先を
|
||||
使用する。DB コネクションはワーカープロセスごとに必要。
|
||||
`COUNTS_EXPORT_DIRECTORY` を同じ永続 POSIX 共有ディレクトリに設定する。
|
||||
省略時は `backend/tmp/counts_exports`。
|
||||
全プロセスで同一の所有者とアクセス権を使用する。
|
||||
複数ホストでは、flock・同一ファイルシステム内の atomic rename・fsync が正しく
|
||||
機能する共有ボリュームが必須。オブジェクトストレージや各コンテナ独立の tmp は不可。
|
||||
ジョブ JSON と完成ファイルが格納されるため、このディレクトリを静的公開しない。
|
||||
|
||||
## API と進捗
|
||||
|
||||
- POST `/counts/export/jobs`:要求検証・実行時点の固定・署名付きジョブトークンの発行。
|
||||
- GET `/counts/export/jobs/:token`:状態、実際の処理済みレコード数、判明している総数。
|
||||
- DELETE `/counts/export/jobs/:token`:取消を永続記録。
|
||||
- GET `/counts/export/files/:token`:生成完了後の署名付きファイルトークンによる取得。
|
||||
|
||||
ジョブ・ファイルのトークンは作成元のセッションに紐付く。
|
||||
既存の BAN・CSRF 制御を使用する。フロントは約800ms間隔で状態を取得する。
|
||||
明細の総数は同じ読み取りトランザクション内の SQL 件数を使用する。
|
||||
queued と running に加え、running 内の phase で aggregating(集計中)と
|
||||
writing(出力中)を区別する。最初の SQL より前に aggregating を記録する。
|
||||
heartbeat_at は監視側の生存時刻、updated_at は生成処理が状態を書いた時刻であり、
|
||||
処理件数と分離する。SQL 中に件数は増やさない。
|
||||
集計の総出力数は生成終了まで未確定とし、不確定進捗として扱う。
|
||||
processed は CSV では実際に生成したデータ行数、JSON のクロス集計では
|
||||
実際に生成した pivot_cell 数。ヘッダーは含めず、小計・合計の物理レコードは含む。
|
||||
通常の集計行だけを数える logical_count_unit=regular_result_rows と区別する。
|
||||
出力中は約250ms間隔と最終レコードで実測件数を保存する。
|
||||
完成状態はファイルの生成完了であり、ブラウザの保存完了を意味しない。
|
||||
|
||||
## 取消・障害・制限
|
||||
|
||||
通常は約250ms間隔の生成チェックポイントで取消を検知する。
|
||||
監視側も取消を確認し、子へ TERM、5秒後も終了していなければ KILL を送る。
|
||||
子の回収後に一時ファイルを削除する。DB 側の実行停止は切断検知等に依存するため、
|
||||
取消 API の受理と DB の即時停止は同義ではない。
|
||||
取消済みへの遷移は生成処理の終了と一時ファイル削除後に行う。
|
||||
既に始まった HTTP 転送やブラウザ側の保存は取り消せない。
|
||||
|
||||
ワーカーが異常終了したジョブは、次のワーカーが実行ロックを取得した際に失敗状態にし、
|
||||
そのジョブの残存生成ファイルを削除する。別の読み取り状態で自動再実行しない。
|
||||
原子的に書き換える状態ファイルとプロセス間ロックにより、二重処理・取消との競合を防ぐ。
|
||||
|
||||
既存の同時実行2、1ファイル128MiB、保存領域512MiB、生成120秒、期限10分を維持する。
|
||||
生成120秒は監視側でも強制する。ワーカー未起動なら登録 API は503を返す。
|
||||
待機中のワーカー消失、15秒以上の heartbeat 停止、150秒以上の待機は失敗とする。
|
||||
ネットワーク切断時も UI が無期限に処理中にならないよう、状態取得は1回15秒で中断し、
|
||||
30秒以上状態を取得できなければ失敗表示へ移る。処理件数を捏造しない。
|
||||
画面からの通信断だけではサーバーの取消完了を保証しない。サーバー側の期限は独立して働く。
|
||||
期限切れのジョブ・ファイル・残存状態書込ファイルは登録時またはワーカーの巡回時に削除する。
|
||||
ワーカーと Web の両方を停止した間は巡回削除も停止する。
|
||||
|
||||
これらの処理は静的確認のみで、実環境のキュー処理・取消・転送・共有ボリュームでの
|
||||
排他動作は未確認。
|
||||
|
||||
## Counts の件数と性能
|
||||
|
||||
通常の Counts 応答では、使用していない全履歴の population_count は計算しない。
|
||||
キーは null、population_count_computed は false とし、0に置き換えない。
|
||||
現在条件の output_count は権限適用前、visible_output_count は権限適用後の
|
||||
通常の集計行数(明細では明細行数)。小計・合計の物理行は含めない。
|
||||
CSV の物理行と JSON の pivot_cell には集計レコードを含むため、両者を混同しない。
|
||||
|
||||
除票境界・参照禁止集合は GROUP BY で識別キー単位にまとめる。
|
||||
投稿の参照判定は、全除票履歴を JSON_CONTAINS / FIND_IN_SET で反復照合せず、
|
||||
保存済みスナップショットから参照IDを JSON_TABLE で展開して照合する。
|
||||
権限条件付きの母集団件数は同一 export インスタンス内で再利用する。
|
||||
多段小計は、認可済み母集団の CTE を MySQL の無制限 LIMIT で実体化させ、
|
||||
各枝が同じ認可判定をやり直すことを避ける。計数は COUNT(DISTINCT v.id) のまま。
|
||||
|
||||
スキーマ上、以下の追加インデックスが候補となる。migration は作成・実行していない。
|
||||
|
||||
- post_versions: (event_type, post_id, version_no)
|
||||
- tag_versions / nico_tag_versions: (event_type, tag_id, version_no)
|
||||
- tag_description_versions: (event_type, tag_id, language_code, version_no)
|
||||
- tag_group_versions: (event_type, tag_group_id, version_no)
|
||||
- material_versions: (event_type, material_id, version_no)
|
||||
- 各履歴テーブル: (event_type, id)。ファイル認可の policy revision 用 MAX(id) の候補。
|
||||
- post_versions 等で未整備の (created_at)。履歴の異動期間を絞る場合の候補。
|
||||
- post_versions: (post_id, created_at, version_no)。日時指定の最新版探索の候補。
|
||||
|
||||
追加は許可後の新規 add_index migration として行い、既存 migration は変更しない。
|
||||
実DBでのインデックス容量・作成時ロック・実行計画・時間は未確認。
|
||||
索引追加なしでの最大データ量に対する応答時間は保証しない。
|
||||
@@ -0,0 +1,26 @@
|
||||
# Counts の言語別列
|
||||
|
||||
- 列は `languages.created_at ASC, languages.code ASC` の順に生成する。
|
||||
全名称列の後に全別名列を置く。Locale の数によって列を重複させない。
|
||||
- 識別子は `name_ja`、`name_ain`、`aliases_ja`、`aliases_ain` 等とする。
|
||||
言語の名称は Frontend の common 辞書を使う。
|
||||
- 名称は記録済みの言語の主名称、別名はその言語の全別名を使う。
|
||||
特定の文字体系で絞らず、記録済みの script_code 等を保持する。
|
||||
履歴の時点評価と HistoryAccess は従来どおり適用する。
|
||||
- 初期選択には UI ロカールの言語コードを使う。明示的な列・空選択は上書きしない。
|
||||
|
||||
## 旧 URL・API
|
||||
|
||||
- 出力列、子列、列順、ソートの旧 `name_ain_kana` 等は、入力境界で
|
||||
`name_ain` 等へ正規化する。表示・ソートは言語全体の値を対象とするため、
|
||||
従来表示されなかった文字体系の値が表示され、並びが変わる場合がある。
|
||||
- 抽出条件の旧ロカール付きキーは別扱いとし、従来の文字体系限定 SQL を維持する。
|
||||
新規条件候補には出さない。既存条件の表示・編輯時だけ文字体系付きの項目名を示す。
|
||||
利用者が新しい言語単位の項目を選択した場合にだけ、その条件を言語全体へ切り替える。
|
||||
- 新しい条件の解釈を `name_scope=language` で明示する。
|
||||
未指定又は `locale` の旧条件は旧仕様として扱い、例えば `name_ja` を
|
||||
`name_ja_jpan` という入力互換専用キーへ変換する。これにより Jpan 限定を保つ。
|
||||
正規化後の URL は `name_scope=language` と互換専用キーを保存し、再解釈を防ぐ。
|
||||
- 新しい API 利用者が言語単位の条件を送る場合も `name_scope=language` を指定する。
|
||||
指定のない既存 API の `name_ja` / `aliases_ja` 条件は従来の Jpan 限定として扱う。
|
||||
未知の name_scope は 400 とする。JSON・Base64 をクエリに格納しない。
|
||||
@@ -0,0 +1,21 @@
|
||||
[Unit]
|
||||
Description=BTRC Hub Counts export supervisor
|
||||
Wants=network-online.target
|
||||
After=network-online.target
|
||||
StartLimitIntervalSec=0
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=miteruzo
|
||||
WorkingDirectory=/home/miteruzo/projects/btrc-hub/backend
|
||||
Environment=RAILS_ENV=production
|
||||
EnvironmentFile=/etc/btrc-hub/counts-exports.env
|
||||
ExecStart=/usr/bin/env bundle exec rake counts:export_worker
|
||||
Restart=always
|
||||
RestartSec=3
|
||||
KillMode=control-group
|
||||
TimeoutStopSec=150
|
||||
UMask=0077
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
+53
-4
@@ -1,5 +1,41 @@
|
||||
# frontend/AGENTS.md
|
||||
|
||||
## Mandatory security and URL contracts
|
||||
|
||||
- URL クエリへの JSON 格納を禁止する。Base64・16進数等による迂回も禁止。
|
||||
単純な配列は半角スペース区切り、構造化条件は番号付きの個別キーで表す。
|
||||
未指定と明示的な空選択を区別し、自由入力を区切り文字で分割しない。
|
||||
Frontend / Backend で同じ文法・上限・検証を使い、新しい URL 状態の追加前に確認する。
|
||||
- 一度除票されたリソースの除票以前(除票版を含む)の履歴は admin 以外へ公開しない。
|
||||
恢復しても解除しない。すべての後続除票を確定版番号と正しい複合キーで判定する。
|
||||
本文・属性・ID・版番号・関連参照・旧値を API / SSR / キャッシュ / 出力から漏らさない。
|
||||
サーバーが確定した current_user の role のみを認可に使用する。
|
||||
秘匿版も安全な全体件数には含めるが、公開明細・内訳・検索件数とは分離する。
|
||||
Counts では、現在の条件に対する権限除外前の出力行数と権限除外レコード数の
|
||||
数値公開及びその差分による間接的な情報開示を許容する。これは件数だけの例外であり、
|
||||
秘匿された本文・属性・識別情報・集計ラベルを公開する例外ではない。
|
||||
それ以外の経路で秘匿属性を条件・軸・ソート・差分から推測できる実装は禁止する。
|
||||
- 秘匿は React / CSS では実装しない。認可済み API のみを使う。
|
||||
認証・権限変更時は取得中リクエストと履歴キャッシュを破棄し、SSR の管理者データを共有しない。
|
||||
|
||||
## Mandatory Counts data contract
|
||||
|
||||
- Counts のロカール非依存・内表行集合保持はルート AGENTS.md の必須契約に従う。
|
||||
内表行の表示フィルターを作らない。名称・別名列はサーバーの Language コードを使い、
|
||||
未登録列に別言語名を代入しない。列を隠しても取得済みの全言語データは保持する。
|
||||
ソート・コピー・JSON の投影には安定したキーと値を使い、UI 訳語を識別に使わない。
|
||||
|
||||
## Mandatory naming contract
|
||||
|
||||
- ルート `AGENTS.md` の必須命名規約を適用する。非 callable の自前識別子に
|
||||
動詞を使わず、形容詞の状態名、名詞の真偽値には `Flg` を使用する。
|
||||
例:`isLatest` は `latest`、`isAdmin` は `adminFlg`。
|
||||
- React / TanStack Query 等の外部 API 名は維持する。取得した値を自前の状態に
|
||||
保持するときは、必要に応じて分割代入等の境界で別名を付ける。
|
||||
callable の state setter、event handler、callback は動詞名を使用できる。
|
||||
- 型、表示項目キー、URL、API、コピー・エクスポートのフィールド参照を整合させる。
|
||||
旧キーの互換性は共通の入出力境界で処理し、新しい内部状態には旧名を使用しない。
|
||||
|
||||
## Scope
|
||||
|
||||
These rules apply to work under `frontend/`.
|
||||
@@ -33,9 +69,9 @@ npm run lint
|
||||
|
||||
If either command cannot be run or fails, report the exact command and failure.
|
||||
|
||||
Do not create, modify, or run tests unless the user explicitly asks for test
|
||||
work. When the user asks for tests, keep working and rerun them until they
|
||||
pass or the remaining failure is clearly blocked.
|
||||
明示的な個別指示がない限り、テストの追加・修正・削除・実行を一切行わない。
|
||||
fixture、mock、snapshot、テスト補助コード及び既存のテスト差分にも触れない。
|
||||
ビルド・lint・静的確認を使い、未確認事項はそのまま報告する。
|
||||
|
||||
## TypeScript
|
||||
|
||||
@@ -128,6 +164,19 @@ pass or the remaining failure is clearly blocked.
|
||||
|
||||
## Tailwind and UI
|
||||
|
||||
- 原則として、ユーザ向け UI 文言を TSX / TypeScript に直接記述しない。
|
||||
新しい文言は既存の i18n システムで管理し、適切な既存キーを先に確認する。
|
||||
不必要な重複定義を避け、新規機能固有の文言は適切な namespace にまとめる。
|
||||
- 対応 locale の辞書を整合させる。未翻訳項目には既存の日本語 fallback と
|
||||
`_untranslated` 規約を使い、language metadata を維持する。
|
||||
翻訳を捏造せず、未翻訳文字列を正式な翻訳として登録しない。
|
||||
- UI 文言の追加・変更時はビルド、静的検査、実画面の確認等で、辞書と
|
||||
SSR / hydration の整合及び翻訳キーの非露出を検証する。
|
||||
i18n の検証を理由にテスト関連作業を開始しない。
|
||||
- API の内部識別子、パラメータ、ユーザ入力データ、外部 URL は機械的に
|
||||
翻訳しない。ボタン、項目名、CSV ヘッダ、エラー、空状態、ツールチップ、
|
||||
アクセシビリティ用ラベルも UI 文言として扱う。
|
||||
|
||||
- Tailwind scans `src/**/*.{html,js,ts,jsx,tsx,mdx}`.
|
||||
- Use `cn` from `src/lib/utils.ts` for conditional class names and class merging.
|
||||
- In JavaScript, JSX, TypeScript, and TSX, use `cn` from `@/lib/utils`
|
||||
@@ -144,7 +193,7 @@ pass or the remaining failure is clearly blocked.
|
||||
heavy borders, or nested panel decoration unless the surrounding screen
|
||||
already uses them.
|
||||
- Keep operational screens dense and direct; trim explanatory copy and use
|
||||
short Japanese labels that fit the control.
|
||||
short localised labels that fit the control.
|
||||
- Preserve existing Japanese tone and orthography in nearby UI text, including
|
||||
old-kana wording where the file already uses it.
|
||||
- Do not add user-facing copy, helper text, descriptions, notes, tooltips,
|
||||
|
||||
生成ファイル
+15
@@ -11,6 +11,7 @@
|
||||
"dependencies": {
|
||||
"@dnd-kit/core": "^6.3.1",
|
||||
"@dnd-kit/modifiers": "^9.0.0",
|
||||
"@dnd-kit/sortable": "^10.0.0",
|
||||
"@dnd-kit/utilities": "^3.2.2",
|
||||
"@fontsource-variable/noto-sans-jp": "^5.2.9",
|
||||
"@mdx-js/react": "^3.1.1",
|
||||
@@ -574,6 +575,20 @@
|
||||
"react": ">=16.8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@dnd-kit/sortable": {
|
||||
"version": "10.0.0",
|
||||
"resolved": "https://registry.npmjs.org/@dnd-kit/sortable/-/sortable-10.0.0.tgz",
|
||||
"integrity": "sha512-+xqhmIIzvAYMGfBYYnbKuNicfSsk4RksY2XdmJhT+HAC01nix6fHCztU68jooFiMUB01Ky3F0FyOvhG/BZrWkg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@dnd-kit/utilities": "^3.2.2",
|
||||
"tslib": "^2.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@dnd-kit/core": "^6.3.0",
|
||||
"react": ">=16.8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@dnd-kit/utilities": {
|
||||
"version": "3.2.2",
|
||||
"resolved": "https://registry.npmjs.org/@dnd-kit/utilities/-/utilities-3.2.2.tgz",
|
||||
|
||||
@@ -18,6 +18,7 @@
|
||||
"dependencies": {
|
||||
"@dnd-kit/core": "^6.3.1",
|
||||
"@dnd-kit/modifiers": "^9.0.0",
|
||||
"@dnd-kit/sortable": "^10.0.0",
|
||||
"@dnd-kit/utilities": "^3.2.2",
|
||||
"@fontsource-variable/noto-sans-jp": "^5.2.9",
|
||||
"@mdx-js/react": "^3.1.1",
|
||||
|
||||
+4
-2
@@ -5,7 +5,7 @@ import { request as requestHTTPS } from 'node:https'
|
||||
import path from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
|
||||
import { render, renderSitemap } from './dist/server/entry-server.js'
|
||||
import { isCountsPath, render, renderSitemap } from './dist/server/entry-server.js'
|
||||
|
||||
const root = fileURLToPath (new URL ('./dist/client/', import.meta.url))
|
||||
const template = await readFile (path.join (root, 'index.html'), 'utf8')
|
||||
@@ -61,6 +61,8 @@ const server = createServer (async (request, response) => {
|
||||
{
|
||||
const url = new URL (request.url, origin)
|
||||
const pathname = decodeURIComponent (url.pathname)
|
||||
const countsPage = isCountsPath (pathname)
|
||||
if (countsPage) response.setHeader ('X-Robots-Tag', 'noindex')
|
||||
if (pathname === '/sitemap.xml' || pathname.startsWith ('/sitemap-'))
|
||||
{
|
||||
const result = await renderSitemap (url.pathname, origin, apiBaseURL)
|
||||
@@ -108,7 +110,7 @@ const server = createServer (async (request, response) => {
|
||||
return
|
||||
}
|
||||
const html = result?.html == null
|
||||
? template.replace ('<!-- helmet -->', result?.status === 404
|
||||
? template.replace ('<!-- helmet -->', (result?.status === 404 || countsPage)
|
||||
? '<meta name="robots" content="noindex" data-rh="true">' : '')
|
||||
: template
|
||||
.replace ('lang="ja"', `lang="${result.locale}"`)
|
||||
|
||||
+116
-1
@@ -62,6 +62,38 @@ const responseFor = url => {
|
||||
return { locales: [{ code: 'ja', name: '日本語' }, { code: 'ain-Kana', name: 'アイヌ' }] }
|
||||
if (path === '/users/settings') return { theme: 'system', wiki_editor_mode: 'split' }
|
||||
if (path === '/users/theme_slots') return []
|
||||
if (path.startsWith ('/counts/definitions/')) {
|
||||
const resource = path.split ('/')[3]
|
||||
const language = locale === 'ain-Kana' ? 'ain' : 'ja'
|
||||
const names = ['ja', 'ain'].map (code => ({ key: `name_${code}`,
|
||||
label: 'counts:localeName', name_locale: { code, language_code: code } }))
|
||||
const columns = resource === 'posts' ? ['resource_id', 'title']
|
||||
: ['resource_id', `name_${language}`]
|
||||
return { fields: [{ key: 'resource_id', type: 'integer', nullable: false,
|
||||
operators: ['eq', 'ne'] }],
|
||||
output_fields: [{ key: 'resource_id', label: 'counts:columns.resourceId' },
|
||||
...(resource === 'posts' ? [{ key: 'title', label: 'counts:columns.postTitle' }] : names)],
|
||||
name_columns: names, filter_fields: [], column_aliases: [], condition_aliases: [],
|
||||
axes: { latest: ['resource_id'], history: ['resource_id'] }, grains: ['month'],
|
||||
defaults: [
|
||||
{ definition: 'latest', mode: 'summary', params: [
|
||||
{ key: 'rows', value: '' }, { key: 'cols', value: '' }] },
|
||||
...['latest', 'history'].map (definition => ({ definition, mode: 'details', params: [
|
||||
{ key: 'columns', value: columns.join (' ') },
|
||||
{ key: 'sort', value: 'resource_id:asc version_no:asc' }] }))] }
|
||||
}
|
||||
if (path === '/counts/query') {
|
||||
const header = { values: [], kind: 'data', depth: 0, axis_count: 0, label_key: null }
|
||||
return { count: 1, page: 1, limit: 100, group_count: 1,
|
||||
output_count: 1, visible_output_count: 1, excluded_count: 0,
|
||||
excluded_count_scope: 'current_query', execution_at: date,
|
||||
rows: [], cols: [], row_headers: [header], column_headers: [header],
|
||||
column_values: [[]], cells: [{ values: [], count: 1 }], pivot_totals: [],
|
||||
condition_labels: {}, axis_labels: {}, versions: [{ id: 1, resource_id: 1,
|
||||
version_no: 1, latest: true, created_at: date, created_by_user: null,
|
||||
title: 'SSR count', name_ja: 'SSR count', name_ain: 'SSR count',
|
||||
names: [], aliases: [], parents: [], tags: [] }] }
|
||||
}
|
||||
if (path === '/posts') return { posts: [post (locale)], count: 1 }
|
||||
if (path === '/gekanator/posts') return { posts: [post (locale)] }
|
||||
if (path === '/gekanator/questions') return { questions: [] }
|
||||
@@ -300,6 +332,29 @@ after (async () => {
|
||||
if (errors.length) throw new AggregateError (errors, 'SSR teardown failed')
|
||||
})
|
||||
|
||||
test ('counts SSR uses query/definitions, noindex and the common site title', async () => {
|
||||
for (const locale of ['ja', 'ain-Kana']) {
|
||||
for (const resource of ['posts', 'tags']) {
|
||||
const column = resource === 'posts' ? 'title' : (locale === 'ja' ? 'name_ja' : 'name_ain')
|
||||
const query =
|
||||
`definition=history&mode=details&columns=resource_id+${ column }&user_id=4&timezone=UTC`
|
||||
const result = await renderPage (`/${locale}/counts/${resource}?${query}`, origin, apiURL)
|
||||
assert.equal (result.status, 200)
|
||||
assert.match (result.html, /SSR count/)
|
||||
assert.match (result.html, new RegExp (`/${locale}/${resource}/1`))
|
||||
assert.doesNotMatch (result.html, /counts:|columns\.changedAt|fields\.actor/)
|
||||
assert.match (result.head, /<meta[^>]*name="robots"[^>]*content="noindex"/)
|
||||
assert.match (result.head, /<title[^>]*>[^<]*集計[^<]*\|[^<]*<\/title>/)
|
||||
assert.ok (requests.some (request => {
|
||||
const url = new URL (request.url, origin)
|
||||
return url.pathname === '/counts/query' && url.searchParams.get ('resource') === resource
|
||||
&& url.searchParams.get ('user_id') === '4'
|
||||
&& url.searchParams.get ('locale') === locale
|
||||
}))
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
test ('every canonical screen route returns page HTML without JavaScript', async t => {
|
||||
const cookie = await authenticatedCookie ()
|
||||
for (const route of routes) {
|
||||
@@ -582,7 +637,8 @@ test ('hydrate every route with stored browser preferences', {
|
||||
}` }, sessionId)
|
||||
const evaluate = async expression => (await send ('Runtime.evaluate', {
|
||||
expression, returnByValue: true }, sessionId)).result.value
|
||||
for (const route of routes.filter (entry => !entry.handle?.redirectTo)) {
|
||||
for (const route of routes.filter (entry => !entry.handle?.redirectTo
|
||||
&& (process.env.SSR_COUNTS_ONLY !== '1' || entry.path.startsWith ('counts')))) {
|
||||
for (const locale of ['ja', 'ain-Kana']) {
|
||||
await t.test (`hydrate ${locale} ${route.path}`, async () => {
|
||||
const path = route.path.replace (':identifier', 'home').replace (/:id|:title/g, '1')
|
||||
@@ -608,6 +664,65 @@ test ('hydrate every route with stored browser preferences', {
|
||||
})
|
||||
}
|
||||
}
|
||||
for (const width of [375, 1240]) {
|
||||
await send ('Emulation.setDeviceMetricsOverride', {
|
||||
width, height: 900, deviceScaleFactor: 1, mobile: width < 768 }, sessionId)
|
||||
for (const resource of ['posts', 'tags']) {
|
||||
await t.test (`counts layout ${width} ${resource}`, async () => {
|
||||
await send ('Page.navigate', {
|
||||
url: `${httpOrigin}/ja/counts/${resource}` }, sessionId)
|
||||
let ready = false
|
||||
for (let attempt = 0; attempt < 100; ++attempt) {
|
||||
await new Promise (resolve => setTimeout (resolve, 75))
|
||||
try {
|
||||
ready = await evaluate (`document.querySelector('tbody tr')!=null`)
|
||||
} catch {}
|
||||
if (ready) break
|
||||
}
|
||||
assert.ok (ready, 'history query results must render')
|
||||
const layout = await evaluate (`(()=>{
|
||||
const nav=document.getElementById('counts-targets');
|
||||
const main=document.querySelector('main').getBoundingClientRect();
|
||||
return {visible:nav.getClientRects().length>0,
|
||||
right:nav.getBoundingClientRect().right,left:main.left,width:main.width};
|
||||
})()`)
|
||||
assert.ok (width >= 768 ? layout.visible && layout.right <= layout.left
|
||||
: !layout.visible && layout.width >= width - 16,
|
||||
'the sidebar sits beside the table on desktop and collapses on mobile')
|
||||
const toggle = `Array.from(document.querySelectorAll('button'))
|
||||
.find(button=>button.textContent.includes('集計条件'))`
|
||||
assert.equal (await evaluate (`(${toggle}).getAttribute('aria-expanded')`), 'false')
|
||||
await evaluate (`(${toggle}).click()`)
|
||||
const positions = await evaluate (`(()=>{
|
||||
const output=document.querySelector('[data-counts-output]').getBoundingClientRect();
|
||||
const filters=document.querySelector('[data-counts-filters]').getBoundingClientRect();
|
||||
return {left:output.left,right:filters.left,top:output.top,bottom:filters.top};
|
||||
})()`)
|
||||
assert.ok (width >= 768 ? positions.left < positions.right
|
||||
: positions.top < positions.bottom, 'output precedes filters in the layout')
|
||||
assert.ok (await evaluate ('document.documentElement.scrollWidth<=innerWidth'))
|
||||
if (resource === 'posts' && process.env.SSR_COUNTS_ONLY === '1') {
|
||||
const { writeFile } = await import ('node:fs/promises')
|
||||
const shot = await send ('Page.captureScreenshot', { format: 'png' }, sessionId)
|
||||
await writeFile (`/tmp/counts-${width}-conditions.png`,
|
||||
Buffer.from (shot.data, 'base64'))
|
||||
}
|
||||
await evaluate (`(${toggle}).click()`)
|
||||
for (const dark of [false, true]) {
|
||||
await evaluate (`document.documentElement.classList.toggle('dark',${dark})`)
|
||||
assert.ok (await evaluate ('document.documentElement.scrollWidth<=innerWidth'),
|
||||
'counts controls must fit the viewport in both themes')
|
||||
if (resource === 'posts' && process.env.SSR_COUNTS_ONLY === '1') {
|
||||
const { writeFile } = await import ('node:fs/promises')
|
||||
const shot = await send ('Page.captureScreenshot', { format: 'png' }, sessionId)
|
||||
await writeFile (`/tmp/counts-${width}-${dark ? 'dark' : 'light'}.png`,
|
||||
Buffer.from (shot.data, 'base64'))
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
if (process.env.SSR_COUNTS_ONLY === '1') return
|
||||
probeActive = true
|
||||
for (const width of [375, 1240]) {
|
||||
await send ('Emulation.setDeviceMetricsOverride', {
|
||||
|
||||
+47
-8
@@ -1,6 +1,6 @@
|
||||
import { useQuery, useQueryClient } from '@tanstack/react-query'
|
||||
import { AnimatePresence, LayoutGroup, MotionConfig, motion } from 'framer-motion'
|
||||
import { createElement, Suspense, lazy, useEffect,
|
||||
import { createElement, Suspense, lazy, useCallback, useEffect, useRef,
|
||||
useLayoutEffect, useMemo, useState } from 'react'
|
||||
import { useTranslation } from 'react-i18next'
|
||||
import { createBrowserRouter,
|
||||
@@ -22,7 +22,8 @@ import { initialiseUserSession } from '@/lib/users'
|
||||
import { prepareHelpPage } from '@/lib/help'
|
||||
import { fetchLocales, isLocaleCode, legacyLocalePath,
|
||||
switchLocalePath, unlocalisedPath } from '@/lib/locales'
|
||||
import { localesKeys, sessionKeys } from '@/lib/queryKeys'
|
||||
import { isAuthorityDependentQuery, localesKeys, sessionKeys } from '@/lib/queryKeys'
|
||||
import { isCountsPath } from '@/lib/routeLocales'
|
||||
import { switchPostFilterLocale } from '@/lib/posts'
|
||||
import { switchWikiLocale } from '@/lib/wiki'
|
||||
import { useLocale } from '@/lib/useLocale'
|
||||
@@ -63,7 +64,10 @@ const lazyPage = <Props extends object,> (
|
||||
})
|
||||
const Deferred = lazy (preload)
|
||||
// A preloaded page must render synchronously on the first hydration pass.
|
||||
const Page = (props: Props) => createElement (loaded ?? Deferred, props)
|
||||
const Page = (props: Props) => {
|
||||
const [Component] = useState (() => loaded ?? Deferred)
|
||||
return createElement (Component, props)
|
||||
}
|
||||
return Object.assign (Page, { preload })
|
||||
}
|
||||
|
||||
@@ -77,6 +81,7 @@ const MaterialListPage = lazyPage (() => import ('@/pages/materials/MaterialList
|
||||
const MaterialNewPage = lazyPage (() => import ('@/pages/materials/MaterialNewPage'))
|
||||
const MaterialSyncSuppressionsPage = lazyPage (
|
||||
() => import ('@/pages/materials/MaterialSyncSuppressionsPage'))
|
||||
const CountsResourcePage = lazyPage (() => import ('@/pages/CountsResourcePage'))
|
||||
const MorePage = lazyPage (() => import ('@/pages/MorePage'))
|
||||
const GekanatorUnavailablePage = lazyPage (() => import ('@/pages/GekanatorUnavailablePage'))
|
||||
const NicoTagListPage = lazyPage (() => import ('@/pages/tags/NicoTagListPage'))
|
||||
@@ -194,6 +199,20 @@ export const pageRoutes = (
|
||||
{ id: 'gekanator', path: 'gekanator', element: <GekanatorRoute user={user}/>,
|
||||
handle: { locale: 'ja', sitemap: ['static'], preload: locale => locale === 'ja'
|
||||
? GekanatorPage.preload () : GekanatorUnavailablePage.preload () } },
|
||||
{ path: 'counts', element: <CountsResourcePage resource="posts"/>,
|
||||
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
|
||||
{ path: 'counts/posts', element: <CountsResourcePage resource="posts"/>,
|
||||
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
|
||||
{ path: 'counts/tags', element: <CountsResourcePage resource="tags"/>,
|
||||
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
|
||||
{ path: 'counts/nico-tags', element: <CountsResourcePage resource="nico_tags"/>,
|
||||
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
|
||||
{ path: 'counts/tag-descriptions', element: <CountsResourcePage resource="tag_descriptions"/>,
|
||||
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
|
||||
{ path: 'counts/tag-groups', element: <CountsResourcePage resource="tag_groups"/>,
|
||||
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
|
||||
{ path: 'counts/materials', element: <CountsResourcePage resource="materials"/>,
|
||||
handle: { preload: CountsResourcePage.preload, prefetch: false, robots: 'noindex' } },
|
||||
{ path: 'more', element: <MorePage/>,
|
||||
handle: { preload: MorePage.preload, sitemap: ['static'] } },
|
||||
{ path: '*', element: <NotFound/>, handle: { notFound: true } }]
|
||||
@@ -206,8 +225,11 @@ const matchPageRoutes = (pathname: string) =>
|
||||
export const preloadPageRoute = async (
|
||||
pathname: string,
|
||||
locale: LocaleCode,
|
||||
intent: 'navigation' | 'prefetch' = 'navigation',
|
||||
): Promise<void> => {
|
||||
if (intent === 'prefetch' && isCountsPath (pathname)) return
|
||||
await Promise.all ((matchPageRoutes (pathname) ?? [])
|
||||
.filter (match => intent === 'navigation' || match.route.handle?.prefetch !== false)
|
||||
.map (match => match.route.handle?.preload?.(locale)))
|
||||
}
|
||||
|
||||
@@ -244,15 +266,32 @@ const RoutedApp = ({ locales }: { locales: AvailableLocale[] }) => {
|
||||
const location = useLocation ()
|
||||
const locale = useLocale ()
|
||||
const route = resolvePageRoute (location.pathname)
|
||||
const robots = route?.route.handle?.robots
|
||||
const robots = isCountsPath (location.pathname) ? 'noindex' : route?.route.handle?.robots
|
||||
const pagePath = unlocalisedPath (location.pathname)
|
||||
const countsPage = pagePath === '/counts' || pagePath.startsWith ('/counts/')
|
||||
const switchLocale = async (targetLocale: LocaleCode): Promise<string> => {
|
||||
const resolve = route?.route.handle?.switchLocale
|
||||
return (resolve != null && route != null)
|
||||
? resolve ({ location, params: route.params, locale, targetLocale })
|
||||
: switchLocalePath (location, targetLocale)
|
||||
}
|
||||
const [user, setUser] = useState<User | null> (
|
||||
const [user, setUserState] = useState<User | null> (
|
||||
() => client.getQueryData<User> (sessionKeys.current) ?? null)
|
||||
const authority = useRef (user)
|
||||
const setUser = useCallback<Dispatch<SetStateAction<User | null>>> (action => {
|
||||
const next = typeof action === 'function' ? action (authority.current) : action
|
||||
if (next?.id !== authority.current?.id || next?.role !== authority.current?.role)
|
||||
{
|
||||
const filters = {
|
||||
predicate: (query: { queryKey: readonly unknown[] }) =>
|
||||
isAuthorityDependentQuery (query.queryKey) }
|
||||
void client.cancelQueries (filters)
|
||||
client.removeQueries (filters)
|
||||
}
|
||||
authority.current = next
|
||||
client.setQueryData (sessionKeys.current, next)
|
||||
setUserState (next)
|
||||
}, [client])
|
||||
const [status, setStatus] = useState (200)
|
||||
const behaviourSettings = useClientBehaviourSettings ()
|
||||
const animationMode = behaviourSettings.animation ?? 'normal'
|
||||
@@ -301,7 +340,7 @@ const RoutedApp = ({ locales }: { locales: AvailableLocale[] }) => {
|
||||
}
|
||||
}) ()
|
||||
return () => { cancelled = true }
|
||||
}, [])
|
||||
}, [setUser])
|
||||
|
||||
useEffect (() => {
|
||||
if (!(user))
|
||||
@@ -350,7 +389,7 @@ const RoutedApp = ({ locales }: { locales: AvailableLocale[] }) => {
|
||||
}>
|
||||
<LayoutGroup>
|
||||
<motion.div
|
||||
layout={animationMode === 'off' ? false : 'position'}
|
||||
layout={(animationMode === 'off' || countsPage) ? false : 'position'}
|
||||
transition={{ layout: appLayoutTransition }}
|
||||
className="relative flex h-dvh w-full flex-col overflow-y-hidden">
|
||||
<TopNav user={user} locales={locales} resolveLocaleSwitch={switchLocale}/>
|
||||
@@ -393,7 +432,7 @@ const LocaleRoute: FC = () => {
|
||||
void i18n.changeLanguage (activeLocale)
|
||||
}, [activeLocale, i18n])
|
||||
|
||||
const redirect = legacyLocalePath (location)
|
||||
const redirect = legacyLocalePath (location, pathname => resolvePageRoute (pathname) != null)
|
||||
if (!(isLocaleCode (locale)) && redirect == null)
|
||||
return <NotFound/>
|
||||
if (isError && data == null)
|
||||
|
||||
@@ -7,6 +7,7 @@ import { localiseTo } from '@/lib/locales'
|
||||
import { useLocale, useLocaleNavigate } from '@/lib/useLocale'
|
||||
import { useOverlayStore } from '@/components/RouteBlockerOverlay'
|
||||
import { prefetchForURL } from '@/lib/prefetchers'
|
||||
import { isCountsPath } from '@/lib/routeLocales'
|
||||
import { useClientBehaviourSettings } from '@/lib/useClientBehaviourSettings'
|
||||
import { cn } from '@/lib/utils'
|
||||
import { currentRenderContext } from '@/lib/renderContext'
|
||||
@@ -53,6 +54,7 @@ export default forwardRef<HTMLAnchorElement, Props> (({
|
||||
const setOverlay = useOverlayStore (s => s.setActive)
|
||||
|
||||
const doPrefetch = async () => {
|
||||
if (isCountsPath (new URL (url).pathname)) return true
|
||||
try
|
||||
{
|
||||
await prefetchForURL (qc, url)
|
||||
@@ -94,6 +96,12 @@ export default forwardRef<HTMLAnchorElement, Props> (({
|
||||
|
||||
ev.preventDefault ()
|
||||
|
||||
if (isCountsPath (new URL (url).pathname))
|
||||
{
|
||||
navigate (path, { replace, ...(state && { state }) })
|
||||
return
|
||||
}
|
||||
|
||||
flushSync (() => {
|
||||
setOverlay (true)
|
||||
})
|
||||
|
||||
@@ -120,6 +120,10 @@ export const menuOutline = (
|
||||
{ ...message ('entertainment'), visible: false, subMenu: [
|
||||
{ ...message ('theatre'), to: '/theatres/1' },
|
||||
{ ...message ('gekanator'), to: '/gekanator' }] },
|
||||
{ ...message ('counts'), visible: false, subMenu: [
|
||||
{ ...message ('posts'), to: '/counts/posts' },
|
||||
{ ...message ('tags'), to: '/counts/tags' },
|
||||
{ ...message ('externalTags'), to: '/counts/nico-tags' }] },
|
||||
{ ...message ('users'), to: '/users/settings', visible: false, subMenu: [
|
||||
{ ...message ('list'), to: '/users', visible: false },
|
||||
{ ...message ('currentUser'), to: `/users/${ user?.id }`, visible: false },
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import { motion } from 'framer-motion'
|
||||
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
import { cn } from '@/lib/utils'
|
||||
|
||||
import type { ReactNode } from 'react'
|
||||
|
||||
|
||||
const AnimatedCollapse = ({ open, children, id, className }: {
|
||||
open: boolean; children: ReactNode; id?: string; className?: string },
|
||||
) => {
|
||||
const motionSettings = useClientMotion ()
|
||||
return <motion.div id={id} initial={false} aria-hidden={!open} inert={!open}
|
||||
animate={{ height: open ? 'auto' : 0, opacity: open ? 1 : 0 }}
|
||||
transition={motionSettings.transition ('collapse')}
|
||||
className={cn ('min-h-0 overflow-hidden', className)}>
|
||||
{children}
|
||||
</motion.div>
|
||||
}
|
||||
|
||||
|
||||
export default AnimatedCollapse
|
||||
@@ -8,13 +8,13 @@ import type { ComponentPropsWithoutRef, FC, FocusEvent } from 'react'
|
||||
const pad = (n: number): string => n.toString ().padStart (2, '0')
|
||||
|
||||
|
||||
const toDateTimeLocalValue = (value: Date) => {
|
||||
const toDateTimeLocalValue = (value: Date, seconds = false) => {
|
||||
const y = value.getFullYear ()
|
||||
const m = pad (value.getMonth () + 1)
|
||||
const day = pad (value.getDate ())
|
||||
const h = pad (value.getHours ())
|
||||
const min = pad (value.getMinutes ())
|
||||
return `${ y }-${ m }-${ day }T${ h }:${ min }`
|
||||
return `${ y }-${ m }-${ day }T${ h }:${ min }${seconds ? `:${pad (value.getSeconds ())}` : ''}`
|
||||
}
|
||||
|
||||
|
||||
@@ -37,15 +37,17 @@ type Props = Omit<ComponentPropsWithoutRef<'input'>, 'onChange'> & {
|
||||
onChange?: (isoUTC: string | null) => void
|
||||
className?: string
|
||||
onBlur?: (ev: FocusEvent<HTMLInputElement>) => void
|
||||
invalid?: boolean }
|
||||
invalid?: boolean
|
||||
precision?: 'minute' | 'second' }
|
||||
|
||||
|
||||
const DateTimeField: FC<Props> = ({ value, onChange, className, onBlur, invalid, ...rest }) => {
|
||||
const DateTimeField: FC<Props> = props => {
|
||||
const { value, onChange, className, onBlur, invalid, precision = 'minute', ...rest } = props
|
||||
const [local, setLocal] = useState ('')
|
||||
|
||||
useEffect (() => {
|
||||
setLocal (value ? toDateTimeLocalValue (new Date (value)) : '')
|
||||
}, [value])
|
||||
setLocal (value ? toDateTimeLocalValue (new Date (value), precision === 'second') : '')
|
||||
}, [value, precision])
|
||||
|
||||
return (
|
||||
<input
|
||||
@@ -61,13 +63,15 @@ const DateTimeField: FC<Props> = ({ value, onChange, className, onBlur, invalid,
|
||||
'focus:ring-2 focus:ring-blue-200']),
|
||||
className)}
|
||||
type="datetime-local"
|
||||
step={60}
|
||||
step={precision === 'second' ? 1 : 60}
|
||||
value={local}
|
||||
aria-invalid={invalid}
|
||||
onChange={ev => {
|
||||
const v = ev.target.value
|
||||
setLocal (v)
|
||||
onChange?.(v ? toMinutePrecisionIsoUtc (v) : null)
|
||||
const iso = v ? (precision === 'second'
|
||||
? new Date (v).toISOString () : toMinutePrecisionIsoUtc (v)) : null
|
||||
onChange?.(iso)
|
||||
}}
|
||||
onBlur={onBlur}/>)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
import { useId, useRef } from 'react'
|
||||
import { CircleHelp } from 'lucide-react'
|
||||
|
||||
import type { ReactNode } from 'react'
|
||||
|
||||
|
||||
const HelpPopover = ({ label, children }: {
|
||||
label: { text: string; language: string }; children: ReactNode },
|
||||
) => {
|
||||
const id = useId ()
|
||||
const trigger = useRef<HTMLButtonElement> (null)
|
||||
const panel = useRef<HTMLDivElement> (null)
|
||||
return <>
|
||||
<button ref={trigger} type="button" popoverTarget={id} aria-label={label.text}
|
||||
lang={label.language} className="inline-flex align-middle"
|
||||
onClick={() => {
|
||||
const rect = trigger.current?.getBoundingClientRect ()
|
||||
if (panel.current != null && rect != null)
|
||||
{
|
||||
panel.current.style.top = `${rect.bottom + 4}px`
|
||||
panel.current.style.left = `${Math.min (rect.left, window.innerWidth - 288)}px`
|
||||
}
|
||||
}}><CircleHelp size={14}/></button>
|
||||
<div ref={panel} id={id} popover="auto" tabIndex={-1}
|
||||
role="dialog" aria-label={label.text} lang={label.language}
|
||||
onToggle={event => {
|
||||
if (event.newState === 'open') panel.current?.focus ()
|
||||
}} className="fixed m-0 w-72 max-w-[90vw] rounded border bg-white p-3
|
||||
text-sm text-slate-900 shadow-lg dark:border-slate-600 dark:bg-slate-900
|
||||
dark:text-slate-100">{children}</div>
|
||||
</>
|
||||
}
|
||||
|
||||
|
||||
export default HelpPopover
|
||||
@@ -0,0 +1,74 @@
|
||||
import { DndContext, DragOverlay, KeyboardSensor, MouseSensor, TouchSensor,
|
||||
closestCenter, useSensor, useSensors } from '@dnd-kit/core'
|
||||
import { SortableContext, arrayMove, horizontalListSortingStrategy,
|
||||
sortableKeyboardCoordinates } from '@dnd-kit/sortable'
|
||||
import { useRef, useState } from 'react'
|
||||
import { createPortal } from 'react-dom'
|
||||
import { CSS } from '@dnd-kit/utilities'
|
||||
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
|
||||
import type { ReactNode } from 'react'
|
||||
|
||||
|
||||
const SortableColumns = ({ ids, fixed = [], name, onChange, children }: {
|
||||
ids: string[]; fixed?: string[]; name: (id: string) => { text: string; language: string }
|
||||
onChange: (ids: string[]) => void; children: (ids: string[]) => ReactNode },
|
||||
) => {
|
||||
const [active, setActive] = useState<string | null> (null)
|
||||
const drag = useRef<{ id: string; ids: string[] } | null> (null)
|
||||
const motion = useClientMotion ()
|
||||
const sensors = useSensors (
|
||||
useSensor (MouseSensor, { activationConstraint: { distance: 4 } }),
|
||||
useSensor (TouchSensor, { activationConstraint: { delay: 350, tolerance: 8 } }),
|
||||
useSensor (KeyboardSensor, { coordinateGetter: sortableKeyboardCoordinates }))
|
||||
const move = (current: string[], from: string, to: string) => {
|
||||
if (fixed.includes (from) || fixed.includes (to)) return current
|
||||
const source = current.indexOf (from)
|
||||
const target = current.indexOf (to)
|
||||
return source < 0 || target < 0 || source === target
|
||||
? current : arrayMove (current, source, target)
|
||||
}
|
||||
const reset = () => {
|
||||
drag.current = null
|
||||
setActive (null)
|
||||
}
|
||||
return <DndContext sensors={sensors} collisionDetection={closestCenter}
|
||||
accessibility={{ announcements: {
|
||||
onDragStart: event => name (String (event.active.id)).text,
|
||||
onDragOver: event => event.over == null ? undefined : name (String (event.over.id)).text,
|
||||
onDragMove: () => undefined, onDragEnd: () => undefined,
|
||||
onDragCancel: () => undefined },
|
||||
screenReaderInstructions: { draggable: active == null ? '' : name (active).text } }}
|
||||
onDragStart={event => {
|
||||
const id = String (event.active.id)
|
||||
drag.current = { id, ids }
|
||||
setActive (id)
|
||||
}} onDragCancel={reset} onDragEnd={event => {
|
||||
const session = drag.current
|
||||
reset ()
|
||||
if (session == null || event.over == null || session.id !== String (event.active.id)
|
||||
|| session.ids.length !== ids.length
|
||||
|| session.ids.some ((id, index) => id !== ids[index])) return
|
||||
const next = move (ids, session.id, String (event.over.id))
|
||||
if (next.some ((id, index) => id !== ids[index])) onChange (next)
|
||||
}}>
|
||||
<SortableContext items={ids} strategy={horizontalListSortingStrategy}>
|
||||
{children (ids)}
|
||||
</SortableContext>
|
||||
{typeof document !== 'undefined' && createPortal (<DragOverlay adjustScale={false}
|
||||
style={{ width: 'max-content', height: 'auto', maxWidth: 'calc(100vw - 2rem)' }}
|
||||
dropAnimation={motion.reduced ? null : {
|
||||
duration: motion.duration ('drop'), easing: motion.easing,
|
||||
keyframes: ({ transform }) => [
|
||||
{ transform: CSS.Translate.toString (transform.initial) },
|
||||
{ transform: CSS.Translate.toString (transform.final) }] }}>
|
||||
{active != null && <div lang={name (active).language}
|
||||
className="w-max max-w-[min(20rem,calc(100vw_-_2rem))] truncate rounded
|
||||
border border-blue-500 bg-white px-2 py-1 text-sm text-slate-900 shadow-lg
|
||||
dark:border-blue-400 dark:bg-slate-800 dark:text-slate-100">{name (active).text}</div>}
|
||||
</DragOverlay>, document.body)}
|
||||
</DndContext>
|
||||
}
|
||||
|
||||
export default SortableColumns
|
||||
@@ -0,0 +1,60 @@
|
||||
import { defaultAnimateLayoutChanges, useSortable } from '@dnd-kit/sortable'
|
||||
import { CSS } from '@dnd-kit/utilities'
|
||||
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
import { cn } from '@/lib/utils'
|
||||
|
||||
import type { CSSProperties, ReactNode } from 'react'
|
||||
|
||||
type Name = { text: string; language: string }
|
||||
|
||||
|
||||
const SortableItem = ({ id, name, fixed, children, moveGroup, handleId, renderShell }: {
|
||||
id: string; name: Name; fixed?: boolean; children: ReactNode
|
||||
moveGroup: (direction: number) => void; handleId: string
|
||||
renderShell?: (shell: { ref: (node: HTMLElement | null) => void
|
||||
style: CSSProperties; handle: ReactNode; dragging: boolean }) => ReactNode },
|
||||
) => {
|
||||
const motion = useClientMotion ()
|
||||
const item = useSortable ({ id, disabled: fixed,
|
||||
transition: { duration: motion.duration ('reorder'), easing: motion.easing },
|
||||
animateLayoutChanges: options => !motion.reduced
|
||||
&& defaultAnimateLayoutChanges ({ ...options, wasDragging: true }) })
|
||||
const handle = fixed ? <span className="w-6 shrink-0"/> : <button type="button"
|
||||
id={handleId} ref={item.setActivatorNodeRef} {...item.attributes} {...item.listeners}
|
||||
aria-label={name.text} lang={name.language}
|
||||
aria-keyshortcuts={renderShell == null
|
||||
? 'Space ArrowUp ArrowDown Enter Escape Alt+ArrowLeft Alt+ArrowRight'
|
||||
: 'Space ArrowLeft ArrowRight Enter Escape'}
|
||||
className="mt-1 w-6 shrink-0 cursor-grab touch-none rounded
|
||||
focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-blue-500
|
||||
active:cursor-grabbing dark:focus-visible:ring-blue-400"
|
||||
onClick={event => event.stopPropagation ()}
|
||||
onKeyDown={event => {
|
||||
if (event.altKey && !item.isDragging
|
||||
&& ['ArrowLeft', 'ArrowRight'].includes (event.key))
|
||||
{
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
moveGroup (event.key === 'ArrowLeft' ? -1 : 1)
|
||||
}
|
||||
else item.listeners?.onKeyDown?.(event)
|
||||
}}><span aria-hidden>⠿</span></button>
|
||||
const style = { transform: CSS.Translate.toString (item.transform), transition: item.transition }
|
||||
if (renderShell != null)
|
||||
return renderShell ({ ref: item.setNodeRef, style, handle, dragging: item.isDragging })
|
||||
return <div ref={item.setNodeRef} style={style}
|
||||
className={cn ('relative max-w-full rounded border border-slate-300 bg-white',
|
||||
'text-slate-900 dark:border-slate-600 dark:bg-slate-900 dark:text-slate-100',
|
||||
item.isOver && !item.isDragging
|
||||
&& 'ring-2 ring-blue-500 dark:ring-blue-400',
|
||||
item.isDragging && 'opacity-30 ring-2 ring-blue-500 dark:ring-blue-400')}>
|
||||
<div className="flex min-w-0 items-start gap-1 px-1">
|
||||
{handle}
|
||||
<div className="min-w-0 flex-1 py-1">{children}</div>
|
||||
</div>
|
||||
</div>
|
||||
}
|
||||
|
||||
|
||||
export default SortableItem
|
||||
@@ -0,0 +1,156 @@
|
||||
import { DndContext, DragOverlay, KeyboardSensor, MouseSensor, TouchSensor,
|
||||
closestCenter, pointerWithin, useDroppable, useSensor, useSensors }
|
||||
from '@dnd-kit/core'
|
||||
import { SortableContext, arrayMove, rectSortingStrategy,
|
||||
sortableKeyboardCoordinates } from '@dnd-kit/sortable'
|
||||
import { useId, useRef, useState } from 'react'
|
||||
import { createPortal } from 'react-dom'
|
||||
import { CSS } from '@dnd-kit/utilities'
|
||||
|
||||
import SortableItem from '@/components/common/SortableItem'
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
import { cn } from '@/lib/utils'
|
||||
|
||||
import type { ReactNode } from 'react'
|
||||
|
||||
type Groups = Record<string, string[]>
|
||||
type Name = { text: string; language: string }
|
||||
|
||||
|
||||
const sameGroups = (left: Groups, right: Groups): boolean =>
|
||||
Object.keys (left).length === Object.keys (right).length
|
||||
&& Object.keys (left).every (key => left[key].length === right[key]?.length
|
||||
&& left[key].every ((id, index) => id === right[key][index]))
|
||||
|
||||
|
||||
const Group = ({ id, children }: { id: string; children: ReactNode }) => {
|
||||
const drop = useDroppable ({ id: `group:${id}` })
|
||||
return <div ref={drop.setNodeRef} className="min-w-0 rounded">{children}</div>
|
||||
}
|
||||
|
||||
|
||||
const SortableList = (
|
||||
{ groups, name, handleName, renderItem, renderGroup, onChange, fixed = [] }: {
|
||||
groups: Groups; name: (id: string) => Name
|
||||
handleName?: (id: string) => Name
|
||||
renderItem: (id: string, group: string) => ReactNode
|
||||
renderGroup: (id: string, content: ReactNode, over: boolean) => ReactNode
|
||||
onChange: (groups: Groups) => void; fixed?: string[] },
|
||||
) => {
|
||||
const drag = useRef<{ id: string; groups: Groups } | null> (null)
|
||||
const scope = useId ()
|
||||
const [active, setActive] = useState<string | null> (null)
|
||||
const [overGroup, setOverGroup] = useState<string | null> (null)
|
||||
const motion = useClientMotion ()
|
||||
const accessibleName = handleName ?? name
|
||||
const sensors = useSensors (
|
||||
useSensor (MouseSensor, { activationConstraint: { distance: 4 } }),
|
||||
useSensor (TouchSensor, { activationConstraint: { delay: 350, tolerance: 8 } }),
|
||||
useSensor (KeyboardSensor, { coordinateGetter: sortableKeyboardCoordinates }))
|
||||
const owner = (id: string, value: Groups) => Object.keys (value)
|
||||
.find (group => value[group].includes (id))
|
||||
const relocate = (value: Groups, id: string, over: string): Groups => {
|
||||
const source = owner (id, value)
|
||||
const target = over.startsWith ('group:') ? over.slice (6) : owner (over, value)
|
||||
if (source == null || target == null || value[target] == null
|
||||
|| fixed.includes (id) || fixed.includes (over))
|
||||
return value
|
||||
if (source === target)
|
||||
{
|
||||
const from = value[source].indexOf (id)
|
||||
const to = over.startsWith ('group:')
|
||||
? value[target].length - 1 : value[target].indexOf (over)
|
||||
if (from === to || to < 0) return value
|
||||
return { ...value, [source]: arrayMove (value[source], from, to) }
|
||||
}
|
||||
const destination = value[target].filter (item => item !== id)
|
||||
const position = destination.indexOf (over)
|
||||
destination.splice (position < 0 ? destination.length : position, 0, id)
|
||||
return { ...value, [source]: value[source].filter (item => item !== id),
|
||||
[target]: destination }
|
||||
}
|
||||
const moveGroup = (id: string, direction: number) => {
|
||||
if (drag.current != null) return
|
||||
const keys = Object.keys (groups)
|
||||
const group = owner (id, groups)
|
||||
const target = keys[keys.indexOf (group ?? '') + direction]
|
||||
if (target != null)
|
||||
{
|
||||
const next = relocate (groups, id, `group:${target}`)
|
||||
if (sameGroups (groups, next)) return
|
||||
onChange (next)
|
||||
requestAnimationFrame (() => document.getElementById (`${scope}-${id}`)?.focus ())
|
||||
}
|
||||
}
|
||||
const reset = () => {
|
||||
drag.current = null
|
||||
setActive (null)
|
||||
setOverGroup (null)
|
||||
}
|
||||
return <DndContext sensors={sensors} collisionDetection={args => {
|
||||
const hits = pointerWithin (args)
|
||||
const group = hits.find (hit => String (hit.id).startsWith ('group:'))
|
||||
const groupName = group?.id.toString ().slice (6)
|
||||
const item = groupName != null && hits.some (hit =>
|
||||
owner (String (hit.id), groups) === groupName)
|
||||
return group != null && !item ? [group] : closestCenter (args)
|
||||
}}
|
||||
accessibility={{ announcements: {
|
||||
onDragStart: event => accessibleName (String (event.active.id)).text,
|
||||
onDragOver: event => event.over == null ? undefined : (
|
||||
event.over.id.toString ().startsWith ('group:') ? undefined
|
||||
: accessibleName (String (event.over.id)).text),
|
||||
onDragMove: () => undefined, onDragEnd: () => undefined,
|
||||
onDragCancel: () => undefined },
|
||||
screenReaderInstructions: {
|
||||
draggable: active == null ? '' : accessibleName (active).text } }}
|
||||
onDragStart={event => {
|
||||
const id = String (event.active.id)
|
||||
drag.current = { id, groups }
|
||||
setActive (id)
|
||||
setOverGroup (owner (id, groups) ?? null)
|
||||
}} onDragOver={event => {
|
||||
const id = event.over?.id.toString ()
|
||||
const group = id == null ? null
|
||||
: (id.startsWith ('group:') ? id.slice (6) : owner (id, groups) ?? null)
|
||||
setOverGroup (previous => previous === group ? previous : group)
|
||||
}} onDragCancel={reset} onDragEnd={event => {
|
||||
const session = drag.current
|
||||
reset ()
|
||||
if (session == null || event.over == null || session.id !== String (event.active.id)
|
||||
|| !sameGroups (session.groups, groups)) return
|
||||
const next = relocate (groups, session.id, String (event.over.id))
|
||||
if (!sameGroups (groups, next)) onChange (next)
|
||||
}}>
|
||||
|
||||
{/* Keep indexes and droppable geometry stable until the drop is committed. */}
|
||||
{Object.entries (groups).map (([group, ids]) => <Group key={group} id={group}>
|
||||
{renderGroup (group, <SortableContext items={ids} strategy={rectSortingStrategy}>
|
||||
<div className="flex min-h-10 flex-wrap items-start gap-1 rounded p-1">
|
||||
{ids.map (id => <SortableItem key={id} id={id}
|
||||
name={accessibleName (id)} fixed={fixed.includes (id)} handleId={`${scope}-${id}`}
|
||||
moveGroup={direction => moveGroup (id, direction)}>
|
||||
{renderItem (id, group)}
|
||||
</SortableItem>)}
|
||||
</div>
|
||||
</SortableContext>, overGroup === group)}
|
||||
</Group>)}
|
||||
{typeof document !== 'undefined' && createPortal (
|
||||
<DragOverlay adjustScale={false}
|
||||
style={{ width: 'max-content', height: 'auto', maxWidth: 'calc(100vw - 2rem)' }}
|
||||
dropAnimation={motion.reduced ? null : {
|
||||
duration: motion.duration ('drop'), easing: motion.easing,
|
||||
keyframes: ({ transform }) => [
|
||||
{ transform: CSS.Translate.toString (transform.initial) },
|
||||
{ transform: CSS.Translate.toString (transform.final) }] }}>
|
||||
{active == null ? null : <div lang={name (active).language}
|
||||
className="w-max max-w-[min(20rem,calc(100vw_-_2rem))] truncate rounded
|
||||
border border-blue-500 bg-white px-2 py-1 text-sm
|
||||
text-slate-900 shadow-lg dark:border-blue-400 dark:bg-slate-800 dark:text-slate-100">
|
||||
{name (active).text}</div>}
|
||||
</DragOverlay>, document.body)}
|
||||
</DndContext>
|
||||
}
|
||||
|
||||
|
||||
export default SortableList
|
||||
@@ -12,6 +12,7 @@ import type { Tag } from '@/types'
|
||||
|
||||
type Props = {
|
||||
includeNico?: boolean
|
||||
includeUnused?: boolean
|
||||
describedBy?: string
|
||||
invalid?: boolean
|
||||
value: string
|
||||
@@ -20,6 +21,7 @@ type Props = {
|
||||
|
||||
const TagInput: FC<Props> = ({
|
||||
includeNico = false,
|
||||
includeUnused = false,
|
||||
describedBy,
|
||||
invalid,
|
||||
value,
|
||||
@@ -57,11 +59,11 @@ const TagInput: FC<Props> = ({
|
||||
{
|
||||
const tags = await apiGet<Tag[]> ('/tags/autocomplete', {
|
||||
locale, signal: controller.signal,
|
||||
params: { q: request.query, nico: includeNico ? '1' : '0' } })
|
||||
params: { q: request.query.replace (/^not:/i, ''), nico: includeNico ? '1' : '0' } })
|
||||
if (active)
|
||||
{
|
||||
setResult ({ locale, includeNico, request,
|
||||
tags: tags.filter (tag => tag.postCount > 0) })
|
||||
tags: includeUnused ? tags : tags.filter (tag => tag.postCount > 0) })
|
||||
}
|
||||
}
|
||||
catch
|
||||
@@ -74,7 +76,7 @@ const TagInput: FC<Props> = ({
|
||||
active = false
|
||||
controller.abort ()
|
||||
}
|
||||
}, [request, locale, includeNico])
|
||||
}, [request, locale, includeNico, includeUnused])
|
||||
|
||||
useEffect (() => {
|
||||
setActiveIndex (-1)
|
||||
@@ -85,7 +87,8 @@ const TagInput: FC<Props> = ({
|
||||
// TODO: TagSearch からのコピペのため,共通化を考へる.
|
||||
const handleTagSelect = (tag: Tag) => {
|
||||
const parts = value?.split (' ')
|
||||
parts[parts.length - 1] = tag.name
|
||||
const negative = /^not:/i.test (parts[parts.length - 1])
|
||||
parts[parts.length - 1] = `${negative ? 'not:' : ''}${tag.name}`
|
||||
setValue (parts.join (' ') + ' ')
|
||||
setRequest ({ query: '' })
|
||||
setActiveIndex (-1)
|
||||
|
||||
@@ -0,0 +1,213 @@
|
||||
import { useEffect, useState } from 'react'
|
||||
|
||||
import CountsConditionValues from '@/components/counts/CountsConditionValues'
|
||||
import DateTimeField from '@/components/common/DateTimeField'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import { countsConditionInputs, countsEventTypes, countsFieldLabel, countsFilterFields,
|
||||
countsSetOperators, useCountsDefinition }
|
||||
from '@/lib/countsDefinitions'
|
||||
import useDialogue from '@/lib/dialogues/useDialogue'
|
||||
import { countsConditionValue } from '@/lib/countsDisplay'
|
||||
import { countsAxisLabel } from '@/lib/countsDefinitions'
|
||||
import { useLocale } from '@/lib/useLocale'
|
||||
import { inputClass } from '@/lib/utils'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import { readCountState, encodeCountState } from '@/lib/queryState'
|
||||
|
||||
import type { CountsResult } from '@/lib/counts'
|
||||
import type { DialogueFormControls } from '@/lib/dialogues/useDialogue'
|
||||
import type { CountsCondition, CountsDefinition } from '@/lib/countsDefinitions'
|
||||
|
||||
|
||||
const ConditionEditor = (
|
||||
{ definition, initial, controls, save }: {
|
||||
definition: CountsDefinition; initial?: CountsCondition
|
||||
controls: DialogueFormControls
|
||||
save: (condition: CountsCondition) => void },
|
||||
) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const fields = definition.filterFields ?? definition.fields
|
||||
const fieldName = (key: string) => (fields.find (item => item.key === key)?.labels
|
||||
?? [countsFieldLabel (key)]).map (label => message (
|
||||
label, fields.find (item => item.key === key)?.labelValues).text).join (' › ')
|
||||
const [condition, setCondition] = useState<CountsCondition> (() => {
|
||||
const next = { ...initial, field: initial?.field ?? 'resource_id',
|
||||
operator: initial?.operator ?? 'eq' }
|
||||
if (next.operator !== 'range') delete next.end
|
||||
return next
|
||||
})
|
||||
const { field, operator } = condition
|
||||
const operators = fields.find (item => item.key === field)?.operators ?? []
|
||||
const inputs = countsConditionInputs (fields.find (item => item.key === field), operator)
|
||||
useEffect (() => {
|
||||
controls.setActions ([{ label: label (initial == null ? 'add' : 'shared:actions.edit'),
|
||||
onSelect: () => save (condition) }])
|
||||
}, [controls, condition, initial, label, save])
|
||||
return <div className="space-y-2">
|
||||
<select value={field} aria-label={message ('field').text}
|
||||
className={inputClass (false, 'h-11 w-full')}
|
||||
onChange={event => {
|
||||
const field = event.target.value
|
||||
const operator = fields.find (item => item.key === field)?.operators[0]
|
||||
if (operator != null) setCondition ({ field, operator })
|
||||
}}>
|
||||
{fields.filter (item => item.operators.length > 0
|
||||
&& (!item.legacy || item.key === field)).map (field => (
|
||||
<option key={field.key} value={field.key}>{fieldName (field.key)}</option>))}
|
||||
</select>
|
||||
<select value={operator} aria-label={message ('operator').text}
|
||||
className={inputClass (false, 'h-11 w-full')}
|
||||
onChange={event => setCondition (previous => {
|
||||
const operator = event.target.value
|
||||
const next = { ...previous, operator }
|
||||
if (operator !== 'range') delete next.end
|
||||
if ((operator === 'like') !== (previous.operator === 'like')) delete next.value
|
||||
const choices = countsConditionInputs (fields.find (item => item.key === field), operator)
|
||||
if (countsSetOperators.includes (operator))
|
||||
{
|
||||
delete next.value
|
||||
delete next.end
|
||||
next.input = choices.includes (previous.input ?? 'single') ? previous.input : choices[0]
|
||||
}
|
||||
else
|
||||
{
|
||||
delete next.values
|
||||
delete next.input
|
||||
}
|
||||
return next
|
||||
})}>
|
||||
{operators.map (key => <option key={key} value={key}>
|
||||
{message (`operators.${key}`).text}</option>)}
|
||||
</select>
|
||||
{inputs.length > 1 && <label className="block space-y-2 text-sm">
|
||||
<span>{label ('counts:pattern.input')}</span>
|
||||
<select value={condition.input ?? inputs[0]}
|
||||
className={inputClass (false, 'h-11 w-full')}
|
||||
onChange={event => setCondition (previous => ({ ...previous,
|
||||
input: event.target.value as CountsCondition['input'] }))}>
|
||||
{inputs.map (input => <option key={input} value={input}>
|
||||
{message (`counts:pattern.${input}`).text}</option>)}
|
||||
</select>
|
||||
</label>}
|
||||
</div>
|
||||
}
|
||||
|
||||
|
||||
const CountsAdditionalConditions = ({ params, update, conditionLabels }: {
|
||||
conditionLabels?: CountsResult['conditionLabels']
|
||||
params: URLSearchParams
|
||||
update: (key: string, value: string) => void },
|
||||
) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const definition = useCountsDefinition ()
|
||||
const dialogue = useDialogue ()
|
||||
const locale = useLocale ()
|
||||
const conditions = readCountState<CountsCondition[]> (params, 'conditions', [])
|
||||
const metadata = definition.filterFields ?? definition.fields
|
||||
const fields = metadata.filter (field => field.operators.length > 0 && !field.legacy)
|
||||
const fieldName = (key: string) => (metadata.find (item => item.key === key)?.labels
|
||||
?? [countsFieldLabel (key)]).map (label => message (
|
||||
label, metadata.find (item => item.key === key)?.labelValues).text).join (' › ')
|
||||
const write = (next: CountsCondition[]) => update (
|
||||
'conditions', encodeCountState ('conditions', next))
|
||||
const replace = (index: number, condition: CountsCondition) => write (
|
||||
conditions.map ((item, position) => position === index ? condition : item))
|
||||
const patch = (index: number, change: Partial<CountsCondition>) =>
|
||||
replace (index, { ...conditions[index], ...change })
|
||||
const edit = (index?: number) => {
|
||||
const initial = index == null ? undefined : conditions[index]
|
||||
const save = (condition: CountsCondition) => {
|
||||
if (index == null) write ([...conditions, condition])
|
||||
else replace (index, condition)
|
||||
}
|
||||
void dialogue.form ({ title: label ('additionalConditions'), footerLayout: 'inline',
|
||||
body: controls => <ConditionEditor definition={definition}
|
||||
initial={initial} controls={controls} save={save}/> })
|
||||
}
|
||||
const valueInput = (condition: CountsCondition, index: number, key: 'value' | 'end') => {
|
||||
const type = countsFilterFields (definition)[condition.field]
|
||||
const aria = message (key === 'end' ? 'through'
|
||||
: (condition.operator === 'range' ? 'from' : 'value'))
|
||||
if (type === 'boolean' || condition.field === 'event_type')
|
||||
{
|
||||
const options: string[] = type === 'boolean' ? ['true', 'false'] : [...countsEventTypes]
|
||||
const namespace = type === 'boolean' ? 'truthValues' : 'events'
|
||||
return <select value={condition[key] == null ? '' : String (condition[key])} disabled={condition.blank}
|
||||
className={inputClass (false, 'h-10 w-36 min-w-0 max-w-full py-1')}
|
||||
aria-label={aria.text}
|
||||
lang={aria.language}
|
||||
onChange={event => patch (index, { [key]: event.target.value })}>
|
||||
<option value=""/>
|
||||
{options.map (value => <option key={value} value={value}
|
||||
lang={message (`${namespace}.${value}`).language}>
|
||||
{message (`${namespace}.${value}`).text}</option>)}
|
||||
</select>
|
||||
}
|
||||
if (type === 'datetime')
|
||||
return <DateTimeField precision="second" className="h-10 w-48 min-w-0 max-w-full py-1"
|
||||
value={condition[key] == null ? '' : String (condition[key])} disabled={condition.blank}
|
||||
aria-label={aria.text} lang={aria.language}
|
||||
onChange={value => patch (index, { [key]: value ?? '' })}/>
|
||||
return <input value={condition[key] == null ? '' : String (condition[key])} disabled={condition.blank}
|
||||
type={(type === 'integer' || type === 'interval') ? 'number' : 'text'}
|
||||
className={inputClass (false, 'h-10 w-36 min-w-0 max-w-full py-1')}
|
||||
aria-label={aria.text} lang={aria.language}
|
||||
onChange={event => patch (index, { [key]: event.target.value })}/>
|
||||
}
|
||||
return <fieldset className="min-w-0 space-y-1">
|
||||
<legend className="flex items-center gap-2">{label ('additionalConditions')}
|
||||
<Button type="button" variant="outline" size="sm" disabled={fields.length === 0}
|
||||
onClick={() => edit ()}>{label ('add')}</Button></legend>
|
||||
{conditions.map ((condition, index) => condition.operator === 'axis_eq' ? (
|
||||
<div key={index} className="flex flex-wrap items-center gap-2 border-b py-2">
|
||||
<span>{countsAxisLabel (condition, definition, locale).text}</span>
|
||||
<span>{label ('operatorSymbols.eq')}</span>
|
||||
<span>{countsConditionValue (condition.value, condition, locale,
|
||||
params.get ('timezone') ?? 'UTC', conditionLabels)}</span>
|
||||
<Button type="button" variant="ghost" size="sm"
|
||||
aria-label={message ('remove').text}
|
||||
onClick={() => write (conditions.filter ((_, position) => position !== index))}>
|
||||
×</Button>
|
||||
</div>) : (
|
||||
<div key={index} className="flex min-w-0 flex-wrap items-center gap-x-2 gap-y-2
|
||||
border-b border-slate-200 py-2 dark:border-slate-700">
|
||||
<Button type="button" variant="ghost" size="sm" onClick={() => edit (index)}
|
||||
aria-label={`${fieldName (condition.field)} ${message (
|
||||
`operators.${condition.operator}`).text}`}
|
||||
className="min-w-0 max-w-full justify-start gap-2 whitespace-normal px-1 text-left">
|
||||
<span className="min-w-0 break-words">{fieldName (condition.field)}</span>
|
||||
{condition.operator !== 'range' && <span aria-hidden="true" className="shrink-0">
|
||||
{label (`operatorSymbols.${condition.operator}`)}</span>}
|
||||
</Button>
|
||||
{countsSetOperators.includes (condition.operator) ? (
|
||||
<CountsConditionValues key={`${condition.field}:${condition.operator}`}
|
||||
condition={condition}
|
||||
field={metadata.find (field => field.key === condition.field)!}
|
||||
definition={definition} params={params}
|
||||
maximum={Math.max (0, 100 - conditions.reduce ((sum, item, position) =>
|
||||
sum + (position === index ? 0 : item.values?.length ?? 0), 0))}
|
||||
change={values => patch (index, { values })}/>) : (
|
||||
<div className="inline-flex min-w-0 max-w-full items-center gap-1">
|
||||
{valueInput (condition, index, 'value')}
|
||||
{condition.operator === 'range' && <>
|
||||
<span aria-hidden="true" className="shrink-0">{label ('operatorSymbols.range')}</span>
|
||||
{valueInput (condition, index, 'end')}
|
||||
</>}
|
||||
</div>)}
|
||||
<div className="inline-flex shrink-0 items-center gap-1">
|
||||
<label className="inline-flex items-center gap-1 text-sm">
|
||||
<input type="checkbox" checked={condition.blank ?? false}
|
||||
onChange={event => patch (index, { blank: event.target.checked })}/>
|
||||
{label ('blank')}</label>
|
||||
<Button type="button" variant="ghost" size="sm" className="px-1"
|
||||
aria-label={message ('remove').text}
|
||||
onClick={() => write (conditions.filter ((_, position) => index !== position))}>
|
||||
×</Button>
|
||||
</div>
|
||||
</div>))}
|
||||
</fieldset>
|
||||
}
|
||||
|
||||
|
||||
export default CountsAdditionalConditions
|
||||
@@ -0,0 +1,98 @@
|
||||
import LocalisedText from '@/components/LocalisedText'
|
||||
import SortableList from '@/components/common/SortableList'
|
||||
import { countsAxisFields, countsFilterFields, countsFieldLabel,
|
||||
useCountsDefinition }
|
||||
from '@/lib/countsDefinitions'
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
import { cn, inputClass } from '@/lib/utils'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import { readCountState, encodeCountState } from '@/lib/queryState'
|
||||
|
||||
import type { CountsAxis } from '@/lib/countsDefinitions'
|
||||
|
||||
|
||||
const CountsAxes = ({ params, update }: {
|
||||
params: URLSearchParams; update: (key: string, value: string) => void },
|
||||
) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const definition = useCountsDefinition ()
|
||||
const motion = useClientMotion ()
|
||||
const available = countsAxisFields (definition, params.get ('definition') === 'history')
|
||||
const rows = readCountState<CountsAxis[]> (params, 'rows', [])
|
||||
const cols = readCountState<CountsAxis[]> (params, 'cols', [])
|
||||
const axes = [...rows, ...cols]
|
||||
const groups = {
|
||||
available: available.map (field => field.key).filter (id => !axes.some (
|
||||
axis => axis.field === id)),
|
||||
rows: rows.map (axis => axis.field), cols: cols.map (axis => axis.field) }
|
||||
const name = (id: string) => {
|
||||
const axis = axes.find (item => item.field === id)
|
||||
return axis == null ? message (available.find (field => field.key === id)?.label ?? 'axis')
|
||||
: message (countsFieldLabel (axis.field))
|
||||
}
|
||||
const commit = (next: Record<string, string[]>) => {
|
||||
for (const group of ['rows', 'cols'] as const)
|
||||
{
|
||||
if (next[group].join (' ') === groups[group].join (' ')) continue
|
||||
const selected = next[group].map (field => axes.find (axis => axis.field === field)
|
||||
?? { field, ...(countsFilterFields (definition)[field] === 'datetime'
|
||||
? { grain: 'month' } : { }) })
|
||||
update (group, encodeCountState (group, selected))
|
||||
}
|
||||
}
|
||||
const place = (id: string, target?: 'rows' | 'cols') => {
|
||||
const next = Object.fromEntries (Object.entries (groups).map (([key, ids]) => [
|
||||
key, ids.filter (value => value !== id)]))
|
||||
if (target != null) next[target].push (id)
|
||||
else next.available.push (id)
|
||||
commit (next)
|
||||
}
|
||||
return <SortableList groups={groups} name={name} onChange={commit}
|
||||
renderGroup={(id, content, over) => <fieldset className="mb-2 min-w-0">
|
||||
<legend>{label (id === 'available' ? 'axisItems' : id)}</legend>
|
||||
<div className={cn ('rounded border dark:border-slate-700',
|
||||
over && 'ring-1 ring-blue-500 dark:ring-blue-400')}>{content}</div>
|
||||
</fieldset>} renderItem={(id, group) => {
|
||||
const axis = axes.find (axis => axis.field === id)
|
||||
return <div className={cn ('flex items-center gap-2 text-sm',
|
||||
group !== 'available' && 'flex-wrap')}>
|
||||
<span className="min-w-0 flex-1 break-words">
|
||||
<LocalisedText language={name (id).language}>{name (id).text}</LocalisedText>
|
||||
</span>
|
||||
{group === 'available' && <span className="ml-auto inline-flex shrink-0 gap-1">
|
||||
{(['rows', 'cols'] as const).map (target => <button key={target} type="button"
|
||||
className="h-7 w-7 rounded-full border border-slate-300 bg-slate-100
|
||||
text-xs text-slate-700 hover:bg-blue-100 hover:text-blue-900
|
||||
focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-blue-500
|
||||
dark:border-slate-600 dark:bg-slate-800 dark:text-slate-100
|
||||
dark:hover:bg-blue-900 dark:hover:text-blue-100 dark:focus-visible:ring-blue-400"
|
||||
style={{ transition: `background-color ${motion.duration ('feedback')}ms ${
|
||||
motion.easing}, color ${motion.duration ('feedback')}ms ${motion.easing}` }}
|
||||
aria-label={message ('axisPlace', {
|
||||
field: name (id).text, target: message (target).text }).text}
|
||||
lang={message ('axisPlace').language} onClick={() => place (id, target)}>
|
||||
{label (target)}</button>)}
|
||||
</span>}
|
||||
{group !== 'available' && <>
|
||||
{countsFilterFields (definition)[id] === 'datetime' && <select
|
||||
className={inputClass (false, 'w-auto py-0')}
|
||||
value={axis?.grain ?? 'month'} aria-label={message ('grain').text}
|
||||
onChange={event => update (group, encodeCountState (group,
|
||||
(group === 'rows' ? rows : cols).map (item => item.field === id
|
||||
? { ...item, grain: event.target.value } : item)))}>
|
||||
{definition.grains.map (grain => <option key={grain} value={grain}
|
||||
lang={message (`grains.${grain}`).language}>
|
||||
{message (`grains.${grain}`).text}</option>)}
|
||||
</select>}
|
||||
<button type="button" aria-label={message ('remove').text}
|
||||
lang={message ('remove').language} className="rounded px-1
|
||||
focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-blue-500
|
||||
dark:focus-visible:ring-blue-400" onClick={() => place (id)}>×</button>
|
||||
</>}
|
||||
</div>
|
||||
}}/>
|
||||
}
|
||||
|
||||
|
||||
export default CountsAxes
|
||||
@@ -0,0 +1,139 @@
|
||||
import { useQuery } from '@tanstack/react-query'
|
||||
import { useEffect, useRef, useState } from 'react'
|
||||
|
||||
import { Button } from '@/components/ui/button'
|
||||
import { countsUseLatest } from '@/lib/counts'
|
||||
import { fetchCountsCandidates } from '@/lib/countsDefinitions'
|
||||
import { countsKeys } from '@/lib/queryKeys'
|
||||
import { useLocale } from '@/lib/useLocale'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
import { inputClass } from '@/lib/utils'
|
||||
|
||||
import type { DialogueFormControls } from '@/lib/dialogues/useDialogue'
|
||||
import type { CountsCandidate, CountsDefinition, CountsFieldMetadata }
|
||||
from '@/lib/countsDefinitions'
|
||||
|
||||
|
||||
const CountsCandidatePicker = (
|
||||
{ field, definition, params, initial, maximum, controls, save }: {
|
||||
field: CountsFieldMetadata; params: URLSearchParams
|
||||
definition: CountsDefinition
|
||||
initial: (string | number | boolean)[]; maximum: number
|
||||
controls: DialogueFormControls; save: (values: string[]) => void },
|
||||
) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const locale = useLocale ()
|
||||
const [selected, setSelected] = useState (() => new Set (initial.map (String)))
|
||||
const [columns, setColumns] = useState (['id', 'name', 'category', 'kind'])
|
||||
const [draft, setDraft] = useState ('')
|
||||
const [search, setSearch] = useState ('')
|
||||
const [category, setCategory] = useState ('')
|
||||
const [page, setPage] = useState (1)
|
||||
const point = useRef (params.get ('execution_at'))
|
||||
const request = new URLSearchParams ({ field: field.key, query: search, page: String (page) })
|
||||
for (const key of ['timezone', 'at', 'execution_at'])
|
||||
if (params.has (key)) request.set (key, params.get (key)!)
|
||||
if (countsUseLatest (params)) request.delete ('at')
|
||||
if (category) request.set ('category', category)
|
||||
const dynamic = field.candidateSource === 'tags'
|
||||
const { data, isFetching: fetching, isError: errorFlg } = useQuery ({
|
||||
queryKey: countsKeys.candidates (request.toString (), locale), enabled: dynamic,
|
||||
queryFn: ({ signal }) => {
|
||||
const pinned = new URLSearchParams (request)
|
||||
if (point.current != null) pinned.set ('execution_at', point.current)
|
||||
return fetchCountsCandidates (pinned, locale, signal)
|
||||
}, retry: false })
|
||||
useEffect (() => {
|
||||
if (data != null) point.current = data.executionAt
|
||||
}, [data])
|
||||
useEffect (() => {
|
||||
controls.setActions ([{ label: label ('counts:pattern.selection'),
|
||||
onSelect: () => save ([...selected]) }])
|
||||
}, [controls, label, save, selected])
|
||||
const fixed: CountsCandidate[] = (field.choices ?? []).map (choice => ({
|
||||
value: choice.value, name: message (choice.label).text }))
|
||||
const rows = dynamic ? data?.rows ?? [] : fixed.filter (row =>
|
||||
row.name?.includes (search))
|
||||
const headings: Record<string, string> = {
|
||||
id: 'columns.resourceId', name: 'columns.name', category: 'common:categoryLabel',
|
||||
kind: 'columns.kind' }
|
||||
const available = dynamic ? ['id', 'name', 'category', 'kind'] : ['name']
|
||||
const visible = available.filter (column => columns.includes (column))
|
||||
const categories = definition.filterFields?.find (field =>
|
||||
field.candidateSource === 'categories')?.choices ?? []
|
||||
const cell = (row: CountsCandidate, column: string) => {
|
||||
if (column === 'category') return row.category == null ? ''
|
||||
: message (`common:categories.${row.category}`).text
|
||||
if (column === 'kind') return row.kind == null ? '' : message (`kinds.${row.kind}`).text
|
||||
return column === 'id' ? row.id : row.name
|
||||
}
|
||||
return <div className="min-w-0 space-y-2">
|
||||
<div className="flex min-w-0 items-center gap-2">
|
||||
<input className={inputClass (false, 'h-9 min-w-0 flex-1')}
|
||||
aria-label={message ('common:search').text} value={draft}
|
||||
onChange={event => setDraft (event.target.value)}
|
||||
onKeyDown={event => {
|
||||
if (event.key !== 'Enter') return
|
||||
event.preventDefault ()
|
||||
setSearch (draft)
|
||||
setPage (1)
|
||||
}}/>
|
||||
<Button type="button" onClick={() => { setSearch (draft); setPage (1) }}>
|
||||
{label ('common:search')}</Button>
|
||||
</div>
|
||||
{dynamic && <select className={inputClass (false, 'h-9 w-full')}
|
||||
aria-label={message ('common:categoryLabel').text} value={category}
|
||||
onChange={event => { setCategory (event.target.value); setPage (1) }}>
|
||||
<option value=""/>
|
||||
{categories.map (choice => <option key={choice.value} value={choice.value}>
|
||||
{message (choice.label).text}</option>)}
|
||||
</select>}
|
||||
<fieldset className="flex flex-wrap items-center gap-2 text-sm">
|
||||
<legend>{label ('counts:pattern.columns')}</legend>
|
||||
{available.map (column => <label key={column} className="inline-flex items-center gap-1">
|
||||
<input type="checkbox" checked={columns.includes (column)} onChange={event =>
|
||||
setColumns (previous => event.target.checked ? [...previous, column]
|
||||
: previous.filter (value => value !== column))}/>{label (headings[column])}
|
||||
</label>)}
|
||||
</fieldset>
|
||||
<p className="text-sm">{label ('counts:pattern.value')}{':'}
|
||||
{label (dynamic ? 'columns.resourceId' : 'common:categoryLabel')}
|
||||
</p>
|
||||
<Button type="button" variant="ghost" size="sm" onClick={() => setSelected (new Set ())}>
|
||||
{label ('shared:actions.reset')}</Button>
|
||||
{errorFlg && <p role="alert">{label ('common:fetchError')}</p>}
|
||||
<div className="max-h-[40dvh] overflow-auto" aria-busy={fetching}>
|
||||
<table className="w-full text-left text-sm">
|
||||
<thead><tr><th className="p-2">{label ('counts:pattern.selection')}</th>
|
||||
{visible.map (column => <th key={column} className="p-2">
|
||||
{label (headings[column])}</th>)}
|
||||
</tr></thead>
|
||||
<tbody>{rows.map (row => <tr key={row.value} className="border-t dark:border-slate-700">
|
||||
<td className="p-2"><input type="checkbox" checked={selected.has (row.value)}
|
||||
aria-label={row.name ?? String (row.id ?? '')}
|
||||
disabled={!selected.has (row.value) && selected.size >= maximum}
|
||||
onChange={event => setSelected (previous => {
|
||||
const next = new Set (previous)
|
||||
if (event.target.checked) next.add (row.value)
|
||||
else next.delete (row.value)
|
||||
return next
|
||||
})}/></td>
|
||||
{visible.map (column => <td key={column} className="break-words p-2">
|
||||
{cell (row, column)}</td>)}
|
||||
</tr>)}</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{dynamic && <div className="flex items-center justify-between gap-2">
|
||||
<Button type="button" variant="outline" disabled={page === 1 || fetching}
|
||||
onClick={() => setPage (previous => previous - 1)}>
|
||||
{label ('common:pagination.previous')}</Button>
|
||||
<span>{page}</span>
|
||||
<Button type="button" variant="outline" disabled={!data?.more || fetching}
|
||||
onClick={() => setPage (previous => previous + 1)}>
|
||||
{label ('common:pagination.next')}</Button>
|
||||
</div>}
|
||||
</div>
|
||||
}
|
||||
|
||||
|
||||
export default CountsCandidatePicker
|
||||
@@ -0,0 +1,88 @@
|
||||
import { useEffect, useLayoutEffect, useRef } from 'react'
|
||||
import { createPortal } from 'react-dom'
|
||||
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
export type CountsMenuAction = 'copy' | 'copyHeaders' | 'row' | 'column' | 'all'
|
||||
|
||||
|
||||
const CountsCellMenu = (
|
||||
{ x, y, act, close }: {
|
||||
x: number
|
||||
y: number
|
||||
act: (action: CountsMenuAction) => void
|
||||
close: () => void },
|
||||
) => {
|
||||
const root = useRef<HTMLDivElement> (null)
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
useLayoutEffect (() => {
|
||||
const menu = root.current
|
||||
if (menu == null) return
|
||||
const rect = menu.getBoundingClientRect ()
|
||||
menu.style.left = `${Math.max (4, Math.min (x, window.innerWidth - rect.width - 4))}px`
|
||||
menu.style.top = `${Math.max (4, Math.min (y, window.innerHeight - rect.height - 4))}px`
|
||||
menu.querySelector ('button')?.focus ({ preventScroll: true })
|
||||
}, [x, y])
|
||||
useEffect (() => {
|
||||
const outside = (event: Event) => {
|
||||
if (!root.current?.contains (event.target as Node)) close ()
|
||||
}
|
||||
const escape = (event: KeyboardEvent) => {
|
||||
if (event.key === 'Escape')
|
||||
{
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
close ()
|
||||
}
|
||||
}
|
||||
document.addEventListener ('pointerdown', outside, true)
|
||||
document.addEventListener ('keydown', escape, true)
|
||||
window.addEventListener ('resize', close)
|
||||
document.addEventListener ('scroll', outside, true)
|
||||
return () => {
|
||||
document.removeEventListener ('pointerdown', outside, true)
|
||||
document.removeEventListener ('keydown', escape, true)
|
||||
window.removeEventListener ('resize', close)
|
||||
document.removeEventListener ('scroll', outside, true)
|
||||
}
|
||||
}, [close])
|
||||
const actions: CountsMenuAction[] = ['copy', 'copyHeaders', 'row', 'column', 'all']
|
||||
return createPortal (
|
||||
<div ref={root} role="menu" aria-label={message ('selection.table').text}
|
||||
className="fixed z-50 min-w-40 rounded border border-slate-400 bg-white
|
||||
py-1 text-sm text-slate-900 shadow-lg dark:border-slate-600
|
||||
dark:bg-slate-800 dark:text-slate-100"
|
||||
style={{ left: x, top: y }} onContextMenu={event => event.preventDefault ()}
|
||||
onPointerDown={event => event.stopPropagation ()}
|
||||
onKeyDown={event => {
|
||||
event.stopPropagation ()
|
||||
if ((event.ctrlKey || event.metaKey) && event.key.toLowerCase () === 'c')
|
||||
{
|
||||
event.preventDefault ()
|
||||
act ('copy')
|
||||
return
|
||||
}
|
||||
const buttons = Array.from (root.current?.querySelectorAll ('button') ?? [])
|
||||
const current = buttons.indexOf (document.activeElement as HTMLButtonElement)
|
||||
if (['ArrowDown', 'ArrowUp', 'Home', 'End'].includes (event.key))
|
||||
{
|
||||
event.preventDefault ()
|
||||
const next = event.key === 'Home' ? 0 : (event.key === 'End' ? buttons.length - 1
|
||||
: (current + (event.key === 'ArrowDown' ? 1 : -1) + buttons.length)
|
||||
% buttons.length)
|
||||
buttons[next]?.focus ()
|
||||
}
|
||||
if (event.key === 'Tab') close ()
|
||||
}}>
|
||||
{actions.map (action => (
|
||||
<button key={action} type="button" role="menuitem"
|
||||
className="block w-full px-3 py-1 text-left hover:bg-slate-200
|
||||
hover:text-slate-900 focus:bg-slate-200 focus:text-slate-900
|
||||
dark:hover:bg-slate-700 dark:hover:text-slate-100
|
||||
dark:focus:bg-slate-700 dark:focus:text-slate-100"
|
||||
onClick={() => act (action)}>{label (`selection.${action}`)}</button>))}
|
||||
</div>, document.body)
|
||||
}
|
||||
|
||||
|
||||
export default CountsCellMenu
|
||||
@@ -0,0 +1,84 @@
|
||||
import { useState } from 'react'
|
||||
|
||||
import CountsCandidatePicker from '@/components/counts/CountsCandidatePicker'
|
||||
import DateTimeField from '@/components/common/DateTimeField'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import useDialogue from '@/lib/dialogues/useDialogue'
|
||||
import { inputClass } from '@/lib/utils'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import type { CountsCondition, CountsDefinition, CountsFieldMetadata }
|
||||
from '@/lib/countsDefinitions'
|
||||
|
||||
|
||||
const CountsConditionValues = (
|
||||
{ condition, field, definition, params, maximum, change }: {
|
||||
condition: CountsCondition; field: CountsFieldMetadata; definition: CountsDefinition
|
||||
params: URLSearchParams; maximum: number
|
||||
change: (values: (string | number | boolean)[]) => void },
|
||||
) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const dialogue = useDialogue ()
|
||||
const [draft, setDraft] = useState ('')
|
||||
const values = condition.values ?? []
|
||||
const pick = () => {
|
||||
void dialogue.form ({ title: label (field.candidateSource === 'tags'
|
||||
? 'counts:pattern.link' : 'counts:pattern.pattern'), size: 'large', footerLayout: 'inline',
|
||||
body: controls => <CountsCandidatePicker field={field} definition={definition}
|
||||
params={params} initial={values} maximum={maximum} controls={controls} save={change}/> })
|
||||
}
|
||||
if (condition.input === 'list' || field.candidateSource === 'tags')
|
||||
return <Button type="button" variant="outline" disabled={condition.blank} onClick={pick}>
|
||||
{label ('counts:pattern.list')}{' ('}{values.length}{')'}</Button>
|
||||
const append = () => {
|
||||
if (draft === '' || values.length >= maximum
|
||||
|| values.some (value => String (value) === draft))
|
||||
return
|
||||
change ([...values, draft])
|
||||
setDraft ('')
|
||||
}
|
||||
const input = () => {
|
||||
if (field.type === 'datetime')
|
||||
return <DateTimeField precision="second" value={draft} disabled={condition.blank}
|
||||
className="h-10 w-48 min-w-0 max-w-full" aria-label={message ('value').text}
|
||||
onChange={value => setDraft (value ?? '')}/>
|
||||
if (field.type === 'boolean' || field.choices != null)
|
||||
{
|
||||
const choices = field.choices ?? ['true', 'false'].map (value => ({
|
||||
value, label: `counts:truthValues.${value}` }))
|
||||
return <select value={draft} disabled={condition.blank}
|
||||
className={inputClass (false, 'h-10 w-36 min-w-0 max-w-full')}
|
||||
aria-label={message ('value').text} onChange={event => setDraft (event.target.value)}>
|
||||
<option value=""/>
|
||||
{choices.map (choice => <option key={choice.value} value={choice.value}>
|
||||
{message (choice.label).text}</option>)}
|
||||
</select>
|
||||
}
|
||||
return <input value={draft} disabled={condition.blank}
|
||||
type={field.type === 'integer' ? 'number' : 'text'} aria-label={message ('value').text}
|
||||
className={inputClass (false, 'h-10 w-36 min-w-0 max-w-full')}
|
||||
onChange={event => setDraft (event.target.value)}/>
|
||||
}
|
||||
const display = (value: string | number | boolean) => {
|
||||
const choice = field.choices?.find (choice => choice.value === String (value))
|
||||
if (choice != null) return message (choice.label).text
|
||||
if (field.type === 'boolean') return message (`truthValues.${value}`).text
|
||||
return String (value)
|
||||
}
|
||||
return <div className="flex min-w-0 max-w-full flex-wrap items-center gap-x-2 gap-y-2 py-1">
|
||||
{values.map ((value, index) => <span key={`${index}:${value}`}
|
||||
className="inline-flex items-center gap-1">
|
||||
{display (value)}<Button type="button" size="sm" variant="ghost"
|
||||
aria-label={message ('remove').text} disabled={condition.blank}
|
||||
onClick={() => change (values.filter ((_, position) => position !== index))}>×</Button>
|
||||
</span>)}
|
||||
{input ()}
|
||||
<Button type="button" variant="outline" disabled={condition.blank || draft === ''
|
||||
|| values.length >= maximum} onClick={append}>{label ('add')}</Button>
|
||||
<Button type="button" variant="ghost" disabled={condition.blank}
|
||||
onClick={() => change ([])}>{label ('shared:actions.reset')}</Button>
|
||||
</div>
|
||||
}
|
||||
|
||||
|
||||
export default CountsConditionValues
|
||||
@@ -0,0 +1,119 @@
|
||||
import { fireEvent, screen } from '@testing-library/react'
|
||||
import { useState } from 'react'
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
|
||||
import CountsControls from '@/components/counts/CountsControls'
|
||||
import CountsFieldTree from '@/components/counts/CountsFieldTree'
|
||||
import { CountsDefinitionContext } from '@/lib/countsDefinitions'
|
||||
import { normaliseCountsParams } from '@/lib/counts'
|
||||
import { readCountState, setCountState, encodeCountState } from '@/lib/queryState'
|
||||
import { postDefinition } from '@/test/fixtures/counts'
|
||||
import { renderWithProviders } from '@/test/render'
|
||||
|
||||
vi.mock ('@/lib/api', async original => ({
|
||||
...await original<typeof import ('@/lib/api')> (), apiGet: vi.fn ().mockResolvedValue ([]) }))
|
||||
|
||||
const appliedParams = (extra: Record<string, string> = {}) => normaliseCountsParams ('posts',
|
||||
new URLSearchParams ({ mode: 'details', timezone: 'UTC', ...extra }), postDefinition)
|
||||
|
||||
const renderControls = (params: URLSearchParams, labelsParams = params.toString ()) => {
|
||||
const execute = vi.fn ()
|
||||
const result = renderWithProviders (
|
||||
<CountsDefinitionContext.Provider value={postDefinition}>
|
||||
<CountsControls resource="posts" applied={params} open setOpen={vi.fn ()}
|
||||
execute={execute} reorder={vi.fn ()} displayJSON={vi.fn ()}
|
||||
displayRowNumbers={vi.fn ()} displayThumbnail={vi.fn ()}
|
||||
conditionLabels={{ tags: { 'internal:7': 'Resolved tag' } }}
|
||||
conditionLabelsParams={labelsParams}/>
|
||||
</CountsDefinitionContext.Provider>, { route: '/ja/counts/posts' })
|
||||
return { ...result, execute }
|
||||
}
|
||||
const submit = () => fireEvent.submit (
|
||||
screen.getByRole ('button', { name: '集計実行' }).closest ('form')!)
|
||||
|
||||
|
||||
describe ('Counts filter draft', () => {
|
||||
it ('uses labels only for the executed condition set', () => {
|
||||
const params = appliedParams ()
|
||||
setCountState (params, 'conditions', encodeCountState ('conditions', [
|
||||
{ field: 'tags', operator: 'axis_eq', value: 'internal:7' }]))
|
||||
const { execute } = renderControls (params)
|
||||
expect (screen.getByText ('Resolved tag')).toBeInTheDocument ()
|
||||
fireEvent.change (screen.getByRole ('textbox', { name: 'Id.' }), { target: { value: '9' } })
|
||||
expect (screen.queryByText ('Resolved tag')).not.toBeInTheDocument ()
|
||||
expect (screen.getByText ('…')).toBeInTheDocument ()
|
||||
expect (screen.queryByText ('internal:7')).not.toBeInTheDocument ()
|
||||
submit ()
|
||||
expect (readCountState (execute.mock.lastCall![0], 'conditions', []))
|
||||
.toEqual ([{ field: 'tags', operator: 'axis_eq', value: 'internal:7' }])
|
||||
})
|
||||
|
||||
it ('removes a visible drill condition from the submitted URL state', () => {
|
||||
const params = appliedParams ()
|
||||
setCountState (params, 'conditions', encodeCountState ('conditions', [
|
||||
{ field: 'created_at', grain: 'month', operator: 'axis_eq', value: '2026-10' }]))
|
||||
const { execute } = renderControls (params)
|
||||
expect (screen.getByText (/令和8年10月/)).toBeInTheDocument ()
|
||||
fireEvent.click (screen.getByRole ('button', { name: '削除' }))
|
||||
submit ()
|
||||
expect (readCountState (execute.mock.lastCall![0], 'conditions', [])).toEqual ([])
|
||||
})
|
||||
|
||||
it ('opens the existing add dialogue without inserting a condition immediately', () => {
|
||||
const { execute } = renderControls (appliedParams ())
|
||||
fireEvent.click (screen.getByRole ('button', { name: '追加' }))
|
||||
expect (screen.getByRole ('dialog')).toBeInTheDocument ()
|
||||
expect (execute).not.toHaveBeenCalled ()
|
||||
expect (document.querySelectorAll ('[role="dialog"] select.h-11').length).toBeGreaterThan (0)
|
||||
expect (screen.queryByText ('pattern.input')).not.toBeInTheDocument ()
|
||||
})
|
||||
|
||||
it ('retains stable references only for surviving search tokens', () => {
|
||||
const params = appliedParams ({ tags: 'alpha beta' })
|
||||
setCountState (params, 'tag_refs', encodeCountState ('tag_refs', [
|
||||
{ name: 'alpha', tag_id: 7 }, { name: 'beta', external_tag_id: 8 }]))
|
||||
const { execute } = renderControls (params)
|
||||
const tags = screen.getByDisplayValue ('alpha beta')
|
||||
fireEvent.change (tags, { target: { value: 'not:beta gamma alpha' } })
|
||||
submit ()
|
||||
expect (readCountState (execute.mock.lastCall![0], 'tag_refs', [])).toEqual ([
|
||||
{ name: 'alpha', tag_id: 7 }, { name: 'not:beta', external_tag_id: 8, negative: true }])
|
||||
fireEvent.change (tags, { target: { value: 'gamma' } })
|
||||
submit ()
|
||||
expect (readCountState (execute.mock.lastCall![0], 'tag_refs', [])).toEqual ([])
|
||||
expect (execute.mock.lastCall![0].get ('tags')).toBe ('gamma')
|
||||
expect (screen.queryByText (/internal:7|external:8/)).not.toBeInTheDocument ()
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
const FieldTree = () => {
|
||||
const [params, setParams] = useState (() => appliedParams ({ columns: 'tags',
|
||||
children: 'tags.name_ja' }))
|
||||
return <CountsDefinitionContext.Provider value={postDefinition}>
|
||||
<CountsFieldTree resource="posts" params={params} update={(key, value) => setParams (previous => {
|
||||
const next = new URLSearchParams (previous)
|
||||
next.set (key, value)
|
||||
return next
|
||||
})}/>
|
||||
<output data-testid="fields">{params.toString ()}</output>
|
||||
</CountsDefinitionContext.Provider>
|
||||
}
|
||||
|
||||
it ('derives mixed parent selection and permits selecting all children again after clearing', () => {
|
||||
renderWithProviders (<FieldTree/>, { route: '/ja/counts/posts' })
|
||||
const parent = screen.getByRole ('checkbox', { name: 'タグ' })
|
||||
expect (parent).toHaveAttribute ('aria-checked', 'mixed')
|
||||
expect (parent).toBeEnabled ()
|
||||
fireEvent.click (parent)
|
||||
expect (parent).toBeChecked ()
|
||||
fireEvent.click (parent)
|
||||
expect (parent).not.toBeChecked ()
|
||||
expect (parent).toBeEnabled ()
|
||||
fireEvent.click (parent)
|
||||
expect (parent).toBeChecked ()
|
||||
const params = new URLSearchParams (screen.getByTestId ('fields').textContent ?? '')
|
||||
expect (params.get ('children')?.split (' ')).toEqual ([
|
||||
'tags.name_ja', 'tags.name_ain', 'tags.aliases_ja', 'tags.aliases_ain'])
|
||||
expect (params.get ('columns')).toBe ('tags')
|
||||
})
|
||||
@@ -0,0 +1,303 @@
|
||||
import { useEffect, useId, useRef, useState } from 'react'
|
||||
|
||||
import CountsAdditionalConditions from '@/components/counts/CountsAdditionalConditions'
|
||||
import CountsAxes from '@/components/counts/CountsAxes'
|
||||
import CountsFieldTree from '@/components/counts/CountsFieldTree'
|
||||
import AnimatedCollapse from '@/components/common/AnimatedCollapse'
|
||||
import DateTimeField from '@/components/common/DateTimeField'
|
||||
import FormField from '@/components/common/FormField'
|
||||
import HelpPopover from '@/components/common/HelpPopover'
|
||||
import TagInput from '@/components/common/TagInput'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import { countsRequestParams, countsUseLatest, fieldsForCounts, normaliseCountsParams, readCountSort,
|
||||
selectedCountChildren, selectedCountColumns } from '@/lib/counts'
|
||||
import { countsNestedIsScalar } from '@/lib/countsDisplay'
|
||||
import { useCountsDefinition } from '@/lib/countsDefinitions'
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
import { inputClass } from '@/lib/utils'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import { readCountState, setCountState, encodeCountState } from '@/lib/queryState'
|
||||
|
||||
import type { ReactNode } from 'react'
|
||||
|
||||
import type { CountsResource, CountsResult } from '@/lib/counts'
|
||||
|
||||
|
||||
const CountsControls = (
|
||||
{ resource, applied, open, setOpen, execute, displayJSON, displayRowNumbers,
|
||||
displayThumbnail, reorder, actions, conditionLabels, conditionLabelsParams,
|
||||
pending = false }: {
|
||||
resource: CountsResource; applied: URLSearchParams; open: boolean
|
||||
conditionLabels?: CountsResult['conditionLabels']
|
||||
conditionLabelsParams?: string
|
||||
pending?: boolean
|
||||
setOpen: (open: boolean) => void
|
||||
execute: (params: URLSearchParams) => void
|
||||
displayJSON: (enabled: boolean) => void
|
||||
displayRowNumbers: (enabled: boolean) => void
|
||||
displayThumbnail: (enabled: boolean) => void
|
||||
reorder: (value: string) => void; actions?: ReactNode },
|
||||
) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const definition = useCountsDefinition ()
|
||||
const id = useId ()
|
||||
const motion = useClientMotion ()
|
||||
const [draft, setDraft] = useState (() => new URLSearchParams (applied))
|
||||
const labelParams = conditionLabelsParams == null
|
||||
? null : new URLSearchParams (conditionLabelsParams)
|
||||
const thumbnail = selectedCountColumns (resource, applied, definition).includes ('thumbnail')
|
||||
const previousApplied = useRef ({ resource, params: new URLSearchParams (applied) })
|
||||
const appliedState = applied.toString ()
|
||||
const mode = draft.get ('mode') ?? 'summary'
|
||||
const columns = selectedCountColumns (resource, draft, definition)
|
||||
const details = mode === 'details'
|
||||
const nestedFlg = details && fieldsForCounts (resource, definition).some (field =>
|
||||
columns.includes (field.key) && field.children != null
|
||||
&& !countsNestedIsScalar (field.children, field.key, draft)
|
||||
&& field.children.some (child => selectedCountChildren (draft)
|
||||
.has (`${field.key}.${child.key}`)))
|
||||
useEffect (() => {
|
||||
const nextApplied = new URLSearchParams (appliedState)
|
||||
const previous = previousApplied.current
|
||||
previousApplied.current = { resource, params: nextApplied }
|
||||
setDraft (draft => {
|
||||
if (previous.resource !== resource
|
||||
|| previous.params.get ('definition') !== nextApplied.get ('definition'))
|
||||
return nextApplied
|
||||
const keys = new Set ([...previous.params.keys (), ...nextApplied.keys ()])
|
||||
const changed = [...keys].filter (key =>
|
||||
previous.params.get (key) !== nextApplied.get (key))
|
||||
const before = selectedCountColumns (resource, previous.params, definition)
|
||||
const after = selectedCountColumns (resource, nextApplied, definition)
|
||||
const selectionChanged = before.filter (column => column !== 'thumbnail').join (' ')
|
||||
!== after.filter (column => column !== 'thumbnail').join (' ')
|
||||
const internal = ['execution_at', 'timezone', 'order', 'nested_json', 'row_numbers']
|
||||
if (changed.some (key => key === 'columns' ? selectionChanged : !internal.includes (key)))
|
||||
return nextApplied
|
||||
// Internal URL enrichment and immediate display changes preserve pending edits.
|
||||
const next = new URLSearchParams (draft)
|
||||
for (const key of changed)
|
||||
{
|
||||
if (key === 'columns')
|
||||
{
|
||||
const columns = selectedCountColumns (resource, draft, definition)
|
||||
.filter (column => column !== 'thumbnail')
|
||||
next.set ('columns', (after.includes ('thumbnail')
|
||||
? ['thumbnail', ...columns] : columns).join (' '))
|
||||
continue
|
||||
}
|
||||
next.delete (key)
|
||||
for (const value of nextApplied.getAll (key)) next.append (key, value)
|
||||
}
|
||||
return next.toString () === draft.toString () ? draft : next
|
||||
})
|
||||
}, [appliedState, resource, definition])
|
||||
const history = draft.get ('definition') === 'history'
|
||||
const update = (key: string, value: string) => setDraft (previous => {
|
||||
const next = new URLSearchParams (previous)
|
||||
setCountState (next, key, value)
|
||||
if (key === 'tags')
|
||||
{
|
||||
const tokens = value.split (/\s+/).filter (token => token
|
||||
&& token.toLowerCase () !== 'type:or')
|
||||
const remaining = [...tokens]
|
||||
const references = readCountState<Record<string, string | number | boolean>[]> (
|
||||
previous, 'tag_refs', []).flatMap (reference => {
|
||||
const name = String (reference.name)
|
||||
const exact = remaining.indexOf (name)
|
||||
const index = exact >= 0 ? exact : remaining.findIndex (token =>
|
||||
token.replace (/^not:/i, '') === name.replace (/^not:/i, ''))
|
||||
if (index < 0) return []
|
||||
const [token] = remaining.splice (index, 1)
|
||||
return [token === name ? reference : { ...reference, name: token,
|
||||
negative: token.toLowerCase ().startsWith ('not:') }]
|
||||
})
|
||||
for (const name of [...next.keys ()])
|
||||
if (name === 'tag_refs' || name.startsWith ('tag_refs.')) next.delete (name)
|
||||
if (references.length > 0)
|
||||
setCountState (next, 'tag_refs', encodeCountState ('tag_refs', references))
|
||||
}
|
||||
if (key === 'limit') next.delete ('page')
|
||||
if (next.toString () === previous.toString ()) return previous
|
||||
if (['rows', 'cols'].includes (key) && next.get ('sort') !== '') next.delete ('sort')
|
||||
if (key !== 'mode') return next
|
||||
const resolved = normaliseCountsParams (resource, next, definition)
|
||||
const axes = [...readCountState<unknown[]> (resolved, 'rows', []),
|
||||
...readCountState<unknown[]> (resolved, 'cols', [])]
|
||||
const allowed = value === 'summary'
|
||||
? ['count', ...axes.map ((_, index) => `a${index}`)]
|
||||
: fieldsForCounts (resource, definition).map (field => field.key)
|
||||
if (readCountSort (resolved.get ('sort')).some (sort => !allowed.includes (sort.key)
|
||||
&& !(value === 'summary' && /^m[0-9]{1,2}$/.test (sort.key))))
|
||||
next.delete ('sort')
|
||||
return normaliseCountsParams (resource, next, definition)
|
||||
})
|
||||
return <form className="shrink-0" onSubmit={event => {
|
||||
event.preventDefault ()
|
||||
if (pending) return
|
||||
const next = new URLSearchParams (draft)
|
||||
next.delete ('page')
|
||||
execute (normaliseCountsParams (resource, next, definition))
|
||||
}}>
|
||||
{!history && <div className="flex flex-wrap items-center gap-2 border-b pb-1
|
||||
dark:border-slate-700">
|
||||
<div className="flex flex-wrap items-center gap-1 text-sm">
|
||||
{label ('referenceAt')}<HelpPopover label={message ('referenceAt')}>
|
||||
{label ('referenceHelp')}</HelpPopover>
|
||||
<div className="flex min-w-0 max-w-full items-center gap-1">
|
||||
<DateTimeField precision="second" disabled={countsUseLatest (draft)}
|
||||
className="min-w-0 flex-1"
|
||||
value={draft.get ('at') ?? ''}
|
||||
aria-label={message ('referenceAt').text} lang={message ('referenceAt').language}
|
||||
onChange={value => update ('at', value ?? '')}/>
|
||||
<label className="inline-flex shrink-0 items-center gap-1">
|
||||
<input type="checkbox" checked={countsUseLatest (draft)}
|
||||
onChange={event => update ('current', event.target.checked ? '1' : '0')}/>
|
||||
{label ('latest')}</label>
|
||||
</div>
|
||||
</div>
|
||||
</div>}
|
||||
<div className="flex flex-wrap items-center gap-2 border-b py-1 dark:border-slate-700">
|
||||
<Button type="submit" size="sm" aria-disabled={pending}
|
||||
className="aria-disabled:opacity-50">{label ('run')}</Button>
|
||||
<Button type="button" size="sm" variant="outline" aria-expanded={open}
|
||||
aria-controls={id} onClick={() => setOpen (!open)}>
|
||||
{label ('conditions')}<span aria-hidden className="inline-block" style={{
|
||||
transform: open ? 'rotate(90deg)' : 'rotate(0deg)',
|
||||
transition: `transform ${motion.duration ('collapse')}ms ${motion.easing}` }}>▸</span>
|
||||
</Button>
|
||||
<select className={inputClass (false, 'w-auto py-1')}
|
||||
aria-label={message ('mode').text} lang={message ('mode').language}
|
||||
value={mode} onChange={event => update ('mode', event.target.value)}>
|
||||
{['summary', 'details'].map (value => <option key={value} value={value}>
|
||||
{message (value).text}</option>)}
|
||||
</select>
|
||||
{actions}
|
||||
</div>
|
||||
<fieldset className="flex flex-wrap items-center gap-x-3 gap-y-1 border-b py-1
|
||||
dark:border-slate-700">
|
||||
<legend className="sr-only">{label ('displaySettings')}</legend>
|
||||
<label className="inline-flex items-center gap-1 text-sm">
|
||||
{label ('fields.limit')}
|
||||
<select value={draft.get ('limit') ?? '100'}
|
||||
aria-label={message ('fields.limit').text}
|
||||
className={inputClass (false, 'w-20 py-1')}
|
||||
onChange={event => update ('limit', event.target.value)}>
|
||||
{[20, 50, 100].map (count => <option key={count} value={count}>
|
||||
{label ('shared:labels.resultsCount', { count })}</option>)}
|
||||
</select>
|
||||
</label>
|
||||
<label className="inline-flex items-center gap-1 text-sm">
|
||||
<input type="checkbox" checked={draft.get ('row_numbers') === '1'}
|
||||
onChange={event => {
|
||||
update ('row_numbers', event.target.checked ? '1' : '0')
|
||||
displayRowNumbers (event.target.checked)
|
||||
}}/>{label ('rowNumbers')}
|
||||
</label>
|
||||
{details && resource === 'posts' && <label
|
||||
className="inline-flex items-center gap-1 text-sm">
|
||||
<input type="checkbox" checked={thumbnail}
|
||||
onChange={event => displayThumbnail (event.target.checked)}/>
|
||||
{label ('columns.thumbnailDisplay')}
|
||||
</label>}
|
||||
{nestedFlg && <label className="inline-flex items-center gap-1 text-sm">
|
||||
<input type="checkbox" checked={draft.get ('nested_json') === '1'}
|
||||
onChange={event => {
|
||||
update ('nested_json', event.target.checked ? '1' : '0')
|
||||
displayJSON (event.target.checked)
|
||||
}}/>{label ('nested.asJSON')}
|
||||
</label>}
|
||||
</fieldset>
|
||||
<AnimatedCollapse id={id} open={open}>
|
||||
<div className="grid max-h-[40dvh] auto-rows-max gap-3 overflow-y-auto overscroll-contain
|
||||
border-b py-2 [direction:ltr] dark:border-slate-700 md:max-h-none md:auto-rows-auto
|
||||
md:grid-cols-2 md:overflow-visible">
|
||||
<div className="min-h-0 min-w-0 md:max-h-[40dvh] md:overflow-y-auto
|
||||
md:overscroll-contain md:[direction:rtl]">
|
||||
<fieldset className="min-w-0" data-counts-output style={{ direction: 'ltr' }}>
|
||||
{mode === 'summary' ? <CountsAxes params={draft} update={update}/>
|
||||
: <>
|
||||
<legend className="mb-1 font-semibold">{label ('output')}</legend>
|
||||
<CountsFieldTree resource={resource} params={draft} update={(key, value) => {
|
||||
update (key, value)
|
||||
if (key === 'order') reorder (value)
|
||||
}}/>
|
||||
</>}
|
||||
</fieldset>
|
||||
</div>
|
||||
<div className="min-h-0 min-w-0 border-t pt-2 dark:border-slate-700
|
||||
md:max-h-[40dvh] md:overflow-y-auto md:overscroll-contain md:border-t-0 md:pt-0"
|
||||
style={{ direction: 'ltr' }}>
|
||||
<fieldset className="min-w-0 space-y-2" data-counts-filters>
|
||||
<legend className="font-semibold">{label ('filters')}</legend>
|
||||
{history && <div className="flex min-w-0 max-w-full items-center gap-1">
|
||||
<DateTimeField precision="second" value={draft.get ('from') ?? ''}
|
||||
aria-label={message ('from').text} lang={message ('from').language}
|
||||
className="min-w-0 flex-1" onChange={value => update ('from', value ?? '')}/>
|
||||
<span className="shrink-0">{label ('periodSeparator')}</span>
|
||||
<DateTimeField precision="second" value={draft.get ('through') ?? ''}
|
||||
aria-label={message ('through').text} lang={message ('through').language}
|
||||
className="min-w-0 flex-1" onChange={value => update ('through', value ?? '')}/>
|
||||
</div>}
|
||||
<div className="flex flex-wrap gap-2">
|
||||
{['resource_id', 'user_id'].map (key => <FormField key={key}
|
||||
label={label (key === 'resource_id' ? 'columns.resourceId'
|
||||
: 'fields.actor')}>
|
||||
{() => <input value={draft.get (key) ?? ''}
|
||||
aria-label={message (key === 'resource_id' ? 'columns.resourceId'
|
||||
: 'fields.actor').text}
|
||||
className={inputClass (false, 'w-28 py-1')}
|
||||
onChange={event => update (key, event.target.value)}/>}
|
||||
</FormField>)}
|
||||
</div>
|
||||
{resource === 'posts' && <FormField label={label ('columns.tags')}>
|
||||
{() => <div className="flex flex-wrap items-center gap-2">
|
||||
<TagInput includeNico includeUnused value={draft.get ('tags') ?? ''}
|
||||
setValue={value => update ('tags', value)}/>
|
||||
<select value={(draft.get ('match') === 'any'
|
||||
|| draft.get ('tags')?.split (/\s+/).some (token =>
|
||||
token.toLowerCase () === 'type:or')) ? 'any' : 'all'}
|
||||
className={inputClass (false, 'h-10 w-auto')}
|
||||
aria-label={message ('columns.tags').text}
|
||||
onChange={event => {
|
||||
const match = event.target.value
|
||||
setDraft (previous => {
|
||||
const next = new URLSearchParams (previous)
|
||||
next.set ('match', match)
|
||||
next.set ('tags', (next.get ('tags') ?? '').split (/\s+/)
|
||||
.filter (token => token.toLowerCase () !== 'type:or').join (' '))
|
||||
next.delete ('page')
|
||||
return next
|
||||
})
|
||||
}}>
|
||||
<option value="all">AND</option><option value="any">OR</option>
|
||||
</select>
|
||||
{draft.get ('tags') && <>
|
||||
<Button type="button" variant="ghost" size="sm"
|
||||
aria-label={message ('remove').text} onClick={() => setDraft (previous => {
|
||||
const next = new URLSearchParams (previous)
|
||||
for (const key of [...next.keys ()])
|
||||
if (['tags', 'tag', 'kind', 'match', 'tag_refs', 'page'].includes (key)
|
||||
|| key.startsWith ('tag_refs.')) next.delete (key)
|
||||
return next
|
||||
})}>×</Button>
|
||||
</>}
|
||||
</div>}
|
||||
</FormField>}
|
||||
<CountsAdditionalConditions params={draft} update={update}
|
||||
conditionLabels={(labelParams != null
|
||||
&& draft.get ('current') === labelParams.get ('current')
|
||||
&& draft.get ('at') === labelParams.get ('at')
|
||||
&& countsRequestParams (draft).toString ()
|
||||
=== countsRequestParams (labelParams).toString ())
|
||||
? conditionLabels : undefined}/>
|
||||
</fieldset>
|
||||
</div>
|
||||
</div>
|
||||
</AnimatedCollapse>
|
||||
</form>
|
||||
}
|
||||
|
||||
|
||||
export default CountsControls
|
||||
@@ -0,0 +1,211 @@
|
||||
import { useEffect, useRef, useState } from 'react'
|
||||
|
||||
import LocalisedText from '@/components/LocalisedText'
|
||||
import { apiDelete, apiGet, apiPost, isApiError, startAPIDownload } from '@/lib/api'
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
import { cn } from '@/lib/utils'
|
||||
|
||||
import type { DialogueFormControls } from '@/lib/dialogues/useDialogue'
|
||||
import type { LocaleCode } from '@/lib/locales'
|
||||
|
||||
export type CountsExportHandle = { cancel: () => Promise<boolean> }
|
||||
type ExportStatus = {
|
||||
status: 'starting' | 'queued' | 'running' | 'cancelling' | 'cancelled' | 'ready' | 'failed'
|
||||
processed: number; total: number | null; token?: string
|
||||
downloadAttempt?: string; downloadStatus?: string; error?: string
|
||||
phase?: 'queued' | 'aggregating' | 'writing'; heartbeatAt?: number; updatedAt?: number }
|
||||
|
||||
|
||||
const CountsExportProgress = (
|
||||
{ payload, locale, handle, controls }: {
|
||||
payload: Record<string, string>; locale: LocaleCode
|
||||
handle: CountsExportHandle; controls: DialogueFormControls },
|
||||
) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const motion = useClientMotion ()
|
||||
const [state, setState] = useState<ExportStatus> ({
|
||||
status: 'starting', processed: 0, total: null })
|
||||
const [cancelError, setCancelError] = useState (false)
|
||||
const terminal = useRef (false)
|
||||
const cancelled = useRef (false)
|
||||
const job = useRef<string | null> (null)
|
||||
const creation = useRef<Promise<{ job: string }> | null> (null)
|
||||
const downloaded = useRef (false)
|
||||
const attempt = useRef<string | null> (null)
|
||||
const downloadDeadline = useRef (0)
|
||||
const releaseDownload = useRef<(() => void) | null> (null)
|
||||
const [downloadError, setDownloadError] = useState<string | null> (null)
|
||||
const pollRequest = useRef<AbortController | null> (null)
|
||||
useEffect (() => {
|
||||
handle.cancel = async () => {
|
||||
if (terminal.current) return true
|
||||
releaseDownload.current?.()
|
||||
cancelled.current = true
|
||||
if (job.current != null)
|
||||
{
|
||||
const controller = new AbortController ()
|
||||
const deadline = setTimeout (() => controller.abort (), 15_000)
|
||||
try
|
||||
{
|
||||
setCancelError (false)
|
||||
const result = await apiDelete<ExportStatus> (
|
||||
`/counts/export/jobs/${encodeURIComponent (job.current)}`, {
|
||||
locale, signal: controller.signal })
|
||||
setState (result)
|
||||
}
|
||||
catch
|
||||
{
|
||||
setCancelError (true)
|
||||
}
|
||||
finally
|
||||
{
|
||||
clearTimeout (deadline)
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
}, [handle, locale])
|
||||
useEffect (() => {
|
||||
let active = true
|
||||
let lastResponse = Date.now ()
|
||||
let timer: ReturnType<typeof setTimeout> | undefined
|
||||
const poll = async (token: string) => {
|
||||
const controller = new AbortController ()
|
||||
pollRequest.current = controller
|
||||
const deadline = setTimeout (() => controller.abort (), 15_000)
|
||||
try
|
||||
{
|
||||
const response = await apiGet<ExportStatus> (
|
||||
`/counts/export/jobs/${encodeURIComponent (token)}`, {
|
||||
locale, signal: controller.signal })
|
||||
if (!active) return
|
||||
const result = (cancelled.current && response.status === 'ready')
|
||||
? await apiDelete<ExportStatus> (
|
||||
`/counts/export/jobs/${encodeURIComponent (token)}`, {
|
||||
locale, signal: controller.signal }) : response
|
||||
if (!active) return
|
||||
lastResponse = Date.now ()
|
||||
setState (result)
|
||||
terminal.current = ['failed', 'cancelled'].includes (result.status)
|
||||
if (result.status === 'ready' && result.token != null && !downloaded.current)
|
||||
{
|
||||
downloaded.current = true
|
||||
const receipt = await apiPost<ExportStatus> (
|
||||
`/counts/export/jobs/${encodeURIComponent (token)}/download`, {}, {
|
||||
locale, signal: controller.signal })
|
||||
if (!active || cancelled.current) return
|
||||
if (receipt.downloadAttempt == null) throw new Error ('Missing download attempt')
|
||||
attempt.current = receipt.downloadAttempt
|
||||
downloadDeadline.current = Date.now () + 30_000
|
||||
releaseDownload.current = startAPIDownload ('/counts/export', result.token, () => {
|
||||
if (active)
|
||||
{
|
||||
terminal.current = true
|
||||
setDownloadError ('failed')
|
||||
}
|
||||
}, new URLSearchParams ({ job: token, attempt: receipt.downloadAttempt }))
|
||||
}
|
||||
if (attempt.current != null && result.downloadAttempt === attempt.current)
|
||||
{
|
||||
if (result.downloadStatus === 'started' && !cancelled.current)
|
||||
{
|
||||
terminal.current = true
|
||||
// Native navigation continues after the dialogue unmounts.
|
||||
releaseDownload.current = null
|
||||
controls.close ()
|
||||
return
|
||||
}
|
||||
if (['failed', 'unavailable'].includes (result.downloadStatus ?? ''))
|
||||
{
|
||||
terminal.current = true
|
||||
setDownloadError (result.downloadStatus!)
|
||||
}
|
||||
}
|
||||
if (attempt.current != null && Date.now () > downloadDeadline.current)
|
||||
{
|
||||
terminal.current = true
|
||||
setDownloadError ('timeout')
|
||||
releaseDownload.current?.()
|
||||
}
|
||||
if (!terminal.current) timer = setTimeout (() => poll (token), 800)
|
||||
}
|
||||
catch (error)
|
||||
{
|
||||
if (active)
|
||||
{
|
||||
if ((isApiError (error) && error.response?.status === 404)
|
||||
|| Date.now () - lastResponse > 30_000)
|
||||
{
|
||||
terminal.current = true
|
||||
if (downloaded.current) setDownloadError ('unavailable')
|
||||
setState (previous => ({ ...previous, status: 'failed' }))
|
||||
return
|
||||
}
|
||||
setCancelError (true)
|
||||
timer = setTimeout (() => poll (token), 2000)
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
clearTimeout (deadline)
|
||||
if (pollRequest.current === controller) pollRequest.current = null
|
||||
}
|
||||
}
|
||||
if (creation.current == null)
|
||||
{
|
||||
const controller = new AbortController ()
|
||||
const deadline = setTimeout (() => controller.abort (), 15_000)
|
||||
creation.current = apiPost<{ job: string }> ('/counts/export/jobs', payload, {
|
||||
locale, signal: controller.signal }).finally (() => clearTimeout (deadline))
|
||||
}
|
||||
creation.current.then (async result => {
|
||||
if (!active) return
|
||||
job.current = result.job
|
||||
if (cancelled.current) await handle.cancel ()
|
||||
await poll (result.job)
|
||||
}).catch (() => {
|
||||
if (active)
|
||||
{
|
||||
terminal.current = true
|
||||
setState (previous => ({ ...previous, status: 'failed' }))
|
||||
}
|
||||
})
|
||||
return () => {
|
||||
active = false
|
||||
if (timer != null) clearTimeout (timer)
|
||||
pollRequest.current?.abort ()
|
||||
releaseDownload.current?.()
|
||||
}
|
||||
}, [handle, locale, payload, controls])
|
||||
useEffect (() => {
|
||||
controls.setActions ([])
|
||||
}, [controls])
|
||||
const percent = state.total == null ? null : (state.total === 0 ? 100
|
||||
: Math.min (100, Math.floor (state.processed / state.total * 100)))
|
||||
const busy = !['ready', 'failed', 'cancelled'].includes (state.status)
|
||||
const phase = message (`exportProgress.${state.status === 'running'
|
||||
? state.phase ?? 'running' : state.status}`)
|
||||
return <div className="space-y-3 py-2" aria-busy={busy}>
|
||||
<p>{label ('exportProgress.format', {
|
||||
format: message (`exportFormats.${payload.format}`).text })}</p>
|
||||
<p role="status" aria-live="polite" className="flex items-center gap-2">
|
||||
{busy && <span aria-hidden="true" className={cn (
|
||||
'size-4 shrink-0 rounded-full border-2 border-current border-r-transparent',
|
||||
!motion.reduced && 'animate-spin')}
|
||||
style={{ animationDuration: `${motion.duration ('collapse') * 4}ms` }}/>}
|
||||
<LocalisedText language={phase.language}>{phase.text}</LocalisedText>
|
||||
</p>
|
||||
<progress className="h-3 w-full" max={100} value={percent ?? undefined}
|
||||
aria-label={phase.text} lang={phase.language}/>
|
||||
<p aria-live="polite">{percent == null
|
||||
? label ('exportProgress.processed', { count: state.processed })
|
||||
: label ('exportProgress.measured', {
|
||||
processed: state.processed, total: state.total ?? 0, percent })}</p>
|
||||
{downloadError != null && <p role="alert">
|
||||
{label (`downloadErrors.${downloadError}`)}</p>}
|
||||
{cancelError && <p role="alert">{label ('common:fetchError')}</p>}
|
||||
</div>
|
||||
}
|
||||
|
||||
export default CountsExportProgress
|
||||
@@ -0,0 +1,137 @@
|
||||
import { useId, useLayoutEffect, useRef, useState } from 'react'
|
||||
|
||||
import AnimatedCollapse from '@/components/common/AnimatedCollapse'
|
||||
import SortableList from '@/components/common/SortableList'
|
||||
import { countChildPaths, fieldsForCounts, normaliseCountsParams, orderedCountFields,
|
||||
selectedCountChildren, selectedCountColumns, selectedCountFields } from '@/lib/counts'
|
||||
import { useCountsDefinition } from '@/lib/countsDefinitions'
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import type { CountField, CountsResource } from '@/lib/counts'
|
||||
|
||||
type BranchProps = {
|
||||
field: CountField; path: string; enabled: boolean
|
||||
selected: Set<string>; params: URLSearchParams
|
||||
select: (path: string, checked: boolean) => void
|
||||
update: (key: string, value: string) => void }
|
||||
|
||||
|
||||
const FieldBranch = ({ field, path, enabled, selected, params, select, update }: BranchProps) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const [open, setOpen] = useState (false)
|
||||
const id = useId ()
|
||||
const motion = useClientMotion ()
|
||||
const input = useRef<HTMLInputElement> (null)
|
||||
const leaves = countChildPaths (field.children ?? [], path, true)
|
||||
const count = leaves.filter (key => selected.has (key)).length
|
||||
const checked = field.children == null ? selected.has (path)
|
||||
: count > 0 && count === leaves.length
|
||||
const partial = count > 0 && count < leaves.length
|
||||
useLayoutEffect (() => {
|
||||
if (input.current != null) input.current.indeterminate = partial
|
||||
}, [partial])
|
||||
return <>
|
||||
<div className="flex items-center gap-1">
|
||||
<label className="inline-flex items-center gap-1">
|
||||
<input ref={input} type="checkbox" checked={checked} disabled={!enabled}
|
||||
aria-checked={partial ? 'mixed' : checked}
|
||||
onChange={event => select (path, event.target.checked)}/>
|
||||
{label (field.label, field.labelValues)}
|
||||
</label>
|
||||
{field.children != null && <button type="button" aria-expanded={open}
|
||||
aria-controls={id} aria-label={message (field.label, field.labelValues).text}
|
||||
lang={message (field.label, field.labelValues).language} onClick={() => setOpen (!open)}
|
||||
className="rounded px-1 focus-visible:outline-none focus-visible:ring-2
|
||||
focus-visible:ring-blue-500 dark:focus-visible:ring-blue-400">
|
||||
<span aria-hidden className="inline-block" style={{
|
||||
transform: open ? 'rotate(90deg)' : 'rotate(0deg)',
|
||||
transition: `transform ${motion.duration ('collapse')}ms ${motion.easing}` }}>▸</span>
|
||||
</button>}
|
||||
</div>
|
||||
{field.children != null && <AnimatedCollapse open={open} id={id}>
|
||||
<div className="ml-2 border-l pl-1 dark:border-slate-700">
|
||||
<FieldGroup fields={field.children} path={path} params={params}
|
||||
enabled={enabled} selected={selected} select={select} update={update}/>
|
||||
</div>
|
||||
</AnimatedCollapse>}
|
||||
</>
|
||||
}
|
||||
|
||||
|
||||
const FieldGroup = ({ fields, path, params, enabled, selected, select, update }: {
|
||||
fields: CountField[]; path: string; params: URLSearchParams
|
||||
enabled: boolean; selected: Set<string>; select: BranchProps['select']
|
||||
update: BranchProps['update'] },
|
||||
) => {
|
||||
const { message } = useUITranslation ('counts')
|
||||
const ordered = orderedCountFields (fields, params, path)
|
||||
const prefix = path ? `${path}.` : ''
|
||||
return <SortableList groups={{ items: ordered.map (field => field.key) }}
|
||||
fixed={path === '' ? ['thumbnail'] : []}
|
||||
name={key => message (fields.find (field => field.key === key)!.label,
|
||||
fields.find (field => field.key === key)!.labelValues)}
|
||||
handleName={key => message ('reorder', {
|
||||
column: message (fields.find (field => field.key === key)!.label,
|
||||
fields.find (field => field.key === key)!.labelValues).text })}
|
||||
renderGroup={(_, content) => content}
|
||||
onChange={groups => {
|
||||
const current = fields.map (field => `${prefix}${field.key}`)
|
||||
const other = (params.get ('order') ?? '').split (' ')
|
||||
.filter (key => key && !current.includes (key))
|
||||
update ('order', [...other, ...groups.items.map (key => `${prefix}${key}`)].join (' '))
|
||||
}} renderItem={key => {
|
||||
const field = fields.find (field => field.key === key)!
|
||||
return <FieldBranch field={field} path={`${prefix}${key}`} enabled={enabled}
|
||||
selected={selected} params={params} select={select} update={update}/>
|
||||
}}/>
|
||||
}
|
||||
|
||||
|
||||
const CountsFieldTree = (
|
||||
{ resource, params, update, fields: suppliedFields,
|
||||
selectionKey = 'columns', parent = '' }: {
|
||||
resource: CountsResource
|
||||
fields?: CountField[]
|
||||
selectionKey?: string
|
||||
parent?: string
|
||||
params: URLSearchParams
|
||||
update: (key: string, value: string) => void },
|
||||
) => {
|
||||
const definition = useCountsDefinition ()
|
||||
const resolved = normaliseCountsParams (resource, params, definition)
|
||||
const fields = suppliedFields ?? fieldsForCounts (resource, definition)
|
||||
const roots = suppliedFields == null ? selectedCountColumns (resource, resolved, definition)
|
||||
: selectedCountFields (fields, resolved, selectionKey, parent).map (field => field.key)
|
||||
const selected = selectedCountChildren (resolved, fieldsForCounts (resource, definition))
|
||||
const rootPath = (key: string) => parent ? `${parent}.${key}` : key
|
||||
for (const root of roots) selected.add (rootPath (root))
|
||||
const select = (path: string, checked: boolean) => {
|
||||
const next = new Set (selected)
|
||||
const paths = countChildPaths (fields, parent)
|
||||
.filter (key => key === path || key.startsWith (`${path}.`))
|
||||
for (const key of paths)
|
||||
{
|
||||
if (checked) next.add (key)
|
||||
else next.delete (key)
|
||||
}
|
||||
const nextParams = new URLSearchParams (resolved)
|
||||
nextParams.set ('children', [...next].filter (key => key.includes ('.')).join (' '))
|
||||
const children = selectedCountChildren (nextParams, fieldsForCounts (resource, definition))
|
||||
update ('children', [...children].join (' '))
|
||||
update (selectionKey, fields.filter (field => field.children == null
|
||||
? next.has (rootPath (field.key)) : field.children.some (child =>
|
||||
children.has (`${rootPath (field.key)}.${child.key}`)))
|
||||
.map (field => field.key).join (' '))
|
||||
}
|
||||
const visibleFields = parent === ''
|
||||
? fields.filter (field => field.key !== 'thumbnail') : fields
|
||||
return (
|
||||
<>
|
||||
<FieldGroup fields={visibleFields} path={parent} params={resolved} enabled selected={selected}
|
||||
select={select} update={update}/>
|
||||
</>)
|
||||
}
|
||||
|
||||
|
||||
export default CountsFieldTree
|
||||
@@ -0,0 +1,126 @@
|
||||
import LocalisedText from '@/components/LocalisedText'
|
||||
import PrefetchLink from '@/components/PrefetchLink'
|
||||
import CountsSortHeader from '@/components/counts/CountsSortHeader'
|
||||
import CountsSelectableTable from '@/components/counts/CountsSelectableTable'
|
||||
import { changeCountSort, readCountSort, selectedCountChildren,
|
||||
writeCountSort, orderedCountFields } from '@/lib/counts'
|
||||
import { countsFieldValue, countsLogicalName, countsNestedClipboardTable, countsNestedExport,
|
||||
countsNestedOrder, countsNestedText, countsNestedIsScalar, countsNestedScalarText }
|
||||
from '@/lib/countsDisplay'
|
||||
import { useLocale } from '@/lib/useLocale'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import type { CountField, CountRecord, CountsResource } from '@/lib/counts'
|
||||
|
||||
|
||||
const CountsNestedTable = (
|
||||
{ rows, fields, path, resource, title, params, update }: {
|
||||
rows: CountRecord[]
|
||||
fields: CountField[]
|
||||
path: string
|
||||
resource: CountsResource
|
||||
title: { text: string; language: string }
|
||||
params: URLSearchParams
|
||||
update: (key: string, value: string) => void },
|
||||
) => {
|
||||
const { message } = useUITranslation ('counts')
|
||||
const locale = useLocale ()
|
||||
const selected = selectedCountChildren (params)
|
||||
const columns = orderedCountFields (fields, params, path)
|
||||
.filter (field => selected.has (`${path}.${field.key}`))
|
||||
const allSorts = readCountSort (params.get ('child_sort'))
|
||||
const sorts = allSorts.filter (sort => sort.key.startsWith (`${path}/`))
|
||||
.map (sort => ({ ...sort, key: sort.key.slice (path.length + 1) }))
|
||||
const ordered = countsNestedOrder (rows, path, params, locale)
|
||||
const scalar = (row: CountRecord, field: string) => {
|
||||
const value = countsFieldValue (row, field)
|
||||
if (value == null) return null
|
||||
if (field === 'tagId' || field === 'tagGroupId')
|
||||
return <PrefetchLink to={field === 'tagId'
|
||||
? `/tags/${encodeURIComponent (String (value))}` : '/wiki'}>
|
||||
{String (value)}</PrefetchLink>
|
||||
if (field === 'id')
|
||||
{
|
||||
if (row.kind === 'external') return String (value)
|
||||
const target = path === 'parents' ? resource : 'tags'
|
||||
return <PrefetchLink to={`/${target}/${encodeURIComponent (String (value))}`}>
|
||||
{String (value)}</PrefetchLink>
|
||||
}
|
||||
const display = countsLogicalName (row, field, locale)
|
||||
const metadata = fields.find (item => item.key === field)
|
||||
const nameLocale = metadata?.nameLocale
|
||||
if (nameLocale != null)
|
||||
{
|
||||
if (metadata?.multipleValues)
|
||||
return <span lang={nameLocale.code} className="whitespace-pre-line">{display.text}</span>
|
||||
return <LocalisedText language={nameLocale.languageCode}
|
||||
className="whitespace-pre-line">{display.text}</LocalisedText>
|
||||
}
|
||||
if (typeof value === 'boolean')
|
||||
return <span aria-label={display.text} lang={display.language}>{value ? '✓' : '✗'}</span>
|
||||
if (display.language != null)
|
||||
return <LocalisedText language={display.language}>{display.text}</LocalisedText>
|
||||
if (field === 'name' && typeof row.languageCode === 'string')
|
||||
return <LocalisedText language={row.languageCode}>{String (value)}</LocalisedText>
|
||||
return display.text
|
||||
}
|
||||
if (countsNestedIsScalar (fields, path, params))
|
||||
return <span className="whitespace-pre-line">
|
||||
{countsNestedScalarText (rows, fields, path, params, locale)}</span>
|
||||
return (
|
||||
<div className="counts-nested w-full min-w-0 max-w-full">
|
||||
{params.get ('nested_json') === '1' ? (
|
||||
<pre data-counts-scroll
|
||||
aria-label={title.text} lang={title.language}
|
||||
className="max-h-72 w-full overflow-y-auto whitespace-pre-wrap
|
||||
[overflow-wrap:anywhere] font-sans text-xs">
|
||||
{JSON.stringify (countsNestedExport (rows, fields, path, params, locale))}
|
||||
</pre>) : (columns.length > 0 && rows.length > 0 && (
|
||||
<div data-counts-scroll
|
||||
className="relative isolate max-h-72 w-full min-w-0 overflow-auto">
|
||||
<CountsSelectableTable family={path} columns={columns.map (field => field.key)}
|
||||
identity={`${path}:${params}`} name={title}
|
||||
copyTable={() => countsNestedClipboardTable (rows, fields, path, params, locale)}
|
||||
className="text-xs">
|
||||
<thead className="sticky top-0 z-10 bg-background text-foreground
|
||||
dark:bg-background dark:text-foreground"><tr>
|
||||
{columns.map (field => (
|
||||
<CountsSortHeader key={field.key} field={field.key}
|
||||
label={message (field.label, field.labelValues)}
|
||||
sorts={sorts} structured={field.children != null}
|
||||
change={(key, additive) => {
|
||||
const next = changeCountSort (sorts, key, additive)
|
||||
.map (sort => ({ ...sort, key: `${path}/${sort.key}` }))
|
||||
update ('child_sort', writeCountSort ([
|
||||
...allSorts.filter (sort => !sort.key.startsWith (`${path}/`)),
|
||||
...next]))
|
||||
}}/>))}
|
||||
</tr></thead>
|
||||
<tbody>{ordered.map ((row, index) => (
|
||||
<tr key={index} className="border-t dark:border-slate-700">
|
||||
{columns.map (field => (
|
||||
<td key={field.key}
|
||||
className="overflow-hidden break-words border-t px-3 py-2
|
||||
align-top dark:border-slate-700"
|
||||
data-counts-copy={
|
||||
field.children
|
||||
? countsNestedText (
|
||||
row[field.key] as CountRecord[] ?? [], field.children,
|
||||
`${path}.${field.key}`, params, locale)
|
||||
: countsLogicalName (row, field.key, locale).text}>
|
||||
{field.children
|
||||
? <CountsNestedTable rows={row[field.key] as CountRecord[] ?? []}
|
||||
fields={field.children} path={`${path}.${field.key}`}
|
||||
resource={resource}
|
||||
title={message (field.label, field.labelValues)}
|
||||
params={params} update={update}/>
|
||||
: scalar (row, field.key)}
|
||||
</td>))}
|
||||
</tr>))}</tbody>
|
||||
</CountsSelectableTable>
|
||||
</div>))}
|
||||
</div>)
|
||||
}
|
||||
|
||||
|
||||
export default CountsNestedTable
|
||||
@@ -0,0 +1,93 @@
|
||||
import { useContext, useEffect, useRef, useState } from 'react'
|
||||
|
||||
import { CountsTableScope, MAX_COUNT_WIDTH, MIN_COUNT_WIDTH,
|
||||
useCountsTableLayout } from '@/lib/countsTableLayout'
|
||||
import { cn } from '@/lib/utils'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
|
||||
const CountsResizeHandle = ({ field, label }: {
|
||||
field: string; label: { text: string; language: string } }) => {
|
||||
const family = useContext (CountsTableScope)
|
||||
const layout = useCountsTableLayout ()
|
||||
const { message } = useUITranslation ('counts')
|
||||
const [active, setActive] = useState (false)
|
||||
const press = useRef<{ x: number; y: number; width: number; armed: boolean
|
||||
timer?: ReturnType<typeof setTimeout> } | null> (null)
|
||||
const saved = useRef<{ cursor: string; userSelect: string } | null> (null)
|
||||
const restore = () => {
|
||||
if (saved.current == null) return
|
||||
document.body.style.cursor = saved.current.cursor
|
||||
document.body.style.userSelect = saved.current.userSelect
|
||||
saved.current = null
|
||||
}
|
||||
const finish = () => {
|
||||
if (press.current?.timer != null) clearTimeout (press.current.timer)
|
||||
press.current = null
|
||||
setActive (false)
|
||||
restore ()
|
||||
}
|
||||
const arm = () => {
|
||||
if (press.current == null) return
|
||||
press.current.armed = true
|
||||
saved.current = { cursor: document.body.style.cursor,
|
||||
userSelect: document.body.style.userSelect }
|
||||
document.body.style.cursor = 'col-resize'
|
||||
document.body.style.userSelect = 'none'
|
||||
setActive (true)
|
||||
}
|
||||
useEffect (() => () => {
|
||||
if (press.current?.timer != null) clearTimeout (press.current.timer)
|
||||
if (saved.current != null)
|
||||
{
|
||||
document.body.style.cursor = saved.current.cursor
|
||||
document.body.style.userSelect = saved.current.userSelect
|
||||
}
|
||||
}, [])
|
||||
const name = message ('resize.column', { column: label.text })
|
||||
return (
|
||||
<span role="separator" tabIndex={0} aria-orientation="vertical"
|
||||
aria-label={name.text} lang={name.language}
|
||||
aria-valuenow={layout.width (family, field)} aria-valuemin={MIN_COUNT_WIDTH}
|
||||
aria-valuemax={MAX_COUNT_WIDTH} title={name.text}
|
||||
data-counts-resize
|
||||
className={cn ('absolute inset-y-0 right-0 z-30 w-3 cursor-col-resize touch-none',
|
||||
'select-none outline-offset-[-2px] focus-visible:outline',
|
||||
'before:absolute before:inset-y-1 before:left-1/2 before:border-l',
|
||||
'before:border-slate-300 dark:before:border-slate-600',
|
||||
'hover:before:border-blue-500 dark:hover:before:border-blue-400',
|
||||
active && 'before:border-blue-500 dark:before:border-blue-400')}
|
||||
onPointerDown={event => {
|
||||
event.stopPropagation ()
|
||||
if (!event.isPrimary || event.button !== 0) return
|
||||
event.preventDefault ()
|
||||
finish ()
|
||||
event.currentTarget.setPointerCapture (event.pointerId)
|
||||
press.current = { x: event.clientX, y: event.clientY,
|
||||
width: layout.width (family, field), armed: false }
|
||||
if (event.pointerType === 'mouse') arm ()
|
||||
else press.current.timer = setTimeout (arm, 400)
|
||||
}}
|
||||
onPointerMove={event => {
|
||||
event.stopPropagation ()
|
||||
const state = press.current
|
||||
if (state == null) return
|
||||
if (state.armed)
|
||||
layout.resize (family, field, state.width + event.clientX - state.x)
|
||||
else if (Math.hypot (event.clientX - state.x, event.clientY - state.y) > 8)
|
||||
finish ()
|
||||
}}
|
||||
onPointerUp={finish} onPointerCancel={finish} onLostPointerCapture={finish}
|
||||
onClick={event => { event.preventDefault (); event.stopPropagation () }}
|
||||
onContextMenu={event => event.preventDefault ()}
|
||||
onKeyDown={event => {
|
||||
event.stopPropagation ()
|
||||
if (event.key !== 'ArrowLeft' && event.key !== 'ArrowRight') return
|
||||
event.preventDefault ()
|
||||
layout.resize (family, field, layout.width (family, field)
|
||||
+ (event.key === 'ArrowLeft' ? -10 : 10))
|
||||
}}/>)
|
||||
}
|
||||
|
||||
|
||||
export default CountsResizeHandle
|
||||
@@ -0,0 +1,173 @@
|
||||
import SortableColumns from '@/components/common/SortableColumns'
|
||||
import LocalisedText from '@/components/LocalisedText'
|
||||
import PrefetchLink from '@/components/PrefetchLink'
|
||||
import CountsRowNumber from '@/components/counts/CountsRowNumber'
|
||||
import CountsNestedTable from '@/components/counts/CountsNestedTable'
|
||||
import CountsSortHeader from '@/components/counts/CountsSortHeader'
|
||||
import CountsResizeHandle from '@/components/counts/CountsResizeHandle'
|
||||
import CountsSelectableTable from '@/components/counts/CountsSelectableTable'
|
||||
import PostThumbnailPreview from '@/components/posts/PostThumbnailPreview'
|
||||
import { changeCountSort, countColumns, countsRequestParams, fieldsForCounts,
|
||||
readCountSort, selectedCountColumns, writeCountSort } from '@/lib/counts'
|
||||
import { countsCellCopy, countsCellText, countsDetailClipboardTable,
|
||||
countsImageURL, countsNestedRows } from '@/lib/countsDisplay'
|
||||
import useDialogue from '@/lib/dialogues/useDialogue'
|
||||
import { useLocale } from '@/lib/useLocale'
|
||||
import { useCountsDefinition } from '@/lib/countsDefinitions'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import type { CountColumn, CountsResource,
|
||||
CountsResult, CountsVersion } from '@/lib/counts'
|
||||
|
||||
|
||||
const CountsResults = (
|
||||
{ resource, data, params, pending, update }: {
|
||||
resource: CountsResource
|
||||
data: CountsResult
|
||||
params: URLSearchParams
|
||||
pending: boolean
|
||||
update: (key: string, value: string) => void },
|
||||
) => {
|
||||
const definition = useCountsDefinition ()
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const locale = useLocale ()
|
||||
const dialogue = useDialogue ()
|
||||
const columns = selectedCountColumns (resource, params, definition)
|
||||
const rowNumbers = params.get ('row_numbers') === '1'
|
||||
const fields = fieldsForCounts (resource, definition)
|
||||
const sorts = readCountSort (params.get ('sort') ?? 'resource_id:asc version_no:asc')
|
||||
const sort = (key: string, additive: boolean) =>
|
||||
update ('sort', writeCountSort (changeCountSort (sorts, key, additive)))
|
||||
const cell = (row: CountsVersion, column: CountColumn) => {
|
||||
const field = fields.find (field => field.key === column)!
|
||||
const value = countsCellText (row, column, locale, params)
|
||||
if (field.children != null)
|
||||
{
|
||||
return <CountsNestedTable rows={countsNestedRows (row, column)}
|
||||
fields={field.children} path={column} resource={resource}
|
||||
title={message (field.label, field.labelValues)} params={params} update={update}/>
|
||||
}
|
||||
if (field.nameLocale != null)
|
||||
{
|
||||
if (field.multipleValues)
|
||||
return <span lang={field.nameLocale.code} className="whitespace-pre-line">{value}</span>
|
||||
return <LocalisedText language={field.nameLocale.languageCode}
|
||||
className="whitespace-pre-line">{value}</LocalisedText>
|
||||
}
|
||||
switch (column)
|
||||
{
|
||||
case 'resource_id':
|
||||
if (resource === 'nico_tags') return row.name ? (
|
||||
<PrefetchLink to={`/tags/nico?${new URLSearchParams ({ name: row.name })}`}>
|
||||
{row.resourceId}</PrefetchLink>) : row.resourceId
|
||||
if (resource === 'tag_groups') return <PrefetchLink
|
||||
to={`/wiki/${encodeURIComponent (`タググループ:${row.name ?? ''}`)}`}>
|
||||
{row.resourceId}</PrefetchLink>
|
||||
return <PrefetchLink to={`/${resource === 'tag_descriptions' ? 'tags' : resource}/${
|
||||
encodeURIComponent (row.resourceId)}`}>
|
||||
{row.resourceId}</PrefetchLink>
|
||||
case 'name':
|
||||
return resource === 'nico_tags' && row.name ? (
|
||||
<PrefetchLink to={`/tags/nico?${new URLSearchParams ({ name: row.name })}`}>
|
||||
{value}</PrefetchLink>) : value
|
||||
case 'thumbnail':
|
||||
{
|
||||
const url = countsImageURL (row.thumbnailBase)
|
||||
return url == null ? null : (
|
||||
<button type="button" className="block" aria-label={message (
|
||||
'shared:labels.thumbnail').text}
|
||||
onClick={event => {
|
||||
const trigger = event.currentTarget
|
||||
void dialogue.form ({
|
||||
title: label ('shared:labels.thumbnail'),
|
||||
cancelText: label ('shared:dialogue.close'),
|
||||
size: 'viewport',
|
||||
body: () => <img src={url} alt={message ('shared:labels.thumbnail').text}
|
||||
referrerPolicy="no-referrer" className="h-full w-full object-contain"/>
|
||||
}).then (() => trigger.focus ({ preventScroll: true }))
|
||||
}}>
|
||||
<PostThumbnailPreview url={url} className="h-10 w-16"
|
||||
alt={message ('shared:labels.thumbnail').text} referrerPolicy="no-referrer"/>
|
||||
</button>)
|
||||
}
|
||||
case 'url':
|
||||
case 'thumbnail_base':
|
||||
return <a href={countsImageURL (value) ?? undefined} target="_blank" rel="noreferrer">
|
||||
{value}</a>
|
||||
case 'latest':
|
||||
return <LocalisedText language={message ('latestValue').language}>
|
||||
{value}</LocalisedText>
|
||||
case 'user':
|
||||
return row.createdByUser?.name ? value : (
|
||||
<LocalisedText language={message (
|
||||
row.createdByUser == null ? 'bot' : 'history:anonymous').language}>
|
||||
{value}</LocalisedText>)
|
||||
case 'event_type':
|
||||
return <LocalisedText language={message (`events.${row.eventType}`).language}>
|
||||
{value}</LocalisedText>
|
||||
case 'category':
|
||||
return <LocalisedText language={message (`common:categories.${row.category}`).language}>
|
||||
{value}</LocalisedText>
|
||||
default: return value
|
||||
}
|
||||
}
|
||||
return <SortableColumns ids={columns} fixed={['thumbnail']}
|
||||
name={key => message (fields.find (field => field.key === key)!.label,
|
||||
fields.find (field => field.key === key)!.labelValues)}
|
||||
onChange={next => {
|
||||
const all = fields.map (field => field.key)
|
||||
const other = (params.get ('order') ?? '').split (' ').filter (key => !all.includes (key))
|
||||
update ('order', [...next, ...all.filter (key => !next.includes (key)), ...other].join (' '))
|
||||
}}>{columns => {
|
||||
const tableColumns = rowNumbers ? ['row_number', ...columns] : columns
|
||||
return (
|
||||
<div className="min-h-0 flex-1 overflow-auto" data-counts-results aria-busy={pending}>
|
||||
{tableColumns.length === 0 ? <p>{label ('nested.noColumns')}</p> : (
|
||||
<CountsSelectableTable family={resource}
|
||||
columns={tableColumns}
|
||||
copyTable={() => countsDetailClipboardTable (data.versions ?? [],
|
||||
columns.map (column => fields.find (field => field.key === column)!),
|
||||
params, locale, rowNumbers ? (data.page - 1) * data.limit + 1 : null)}
|
||||
identity={`${resource}:${countsRequestParams (params)}:${tableColumns.join (' ')}:${
|
||||
(data.versions ?? []).map (row => row.id)}`}
|
||||
className="text-left text-sm">
|
||||
<thead className="sticky top-0 z-20 bg-background text-foreground
|
||||
dark:bg-background dark:text-foreground"><tr>
|
||||
{rowNumbers && <CountsRowNumber/>}
|
||||
{columns.map (column => column === 'thumbnail'
|
||||
? <th key={column} data-counts-heading={message (countColumns[column]).text}
|
||||
className="relative px-2 py-1">
|
||||
<span className="sr-only">{label (countColumns[column])}</span>
|
||||
<CountsResizeHandle field={column} label={message (countColumns[column])}/>
|
||||
</th>
|
||||
: <CountsSortHeader key={column} field={column}
|
||||
label={message (fields.find (field => field.key === column)!.label,
|
||||
fields.find (field => field.key === column)!.labelValues)}
|
||||
dragId={column}
|
||||
sorts={sorts} change={sort} structured={fields.find (
|
||||
field => field.key === column)?.children != null}/>)}
|
||||
</tr></thead>
|
||||
<tbody>
|
||||
{(data.versions ?? []).map ((row, index) => (
|
||||
<tr key={row.id} className="border-b dark:border-slate-700">
|
||||
{rowNumbers && <CountsRowNumber
|
||||
value={(data.page - 1) * data.limit + index + 1}/>}
|
||||
{columns.map (column => (
|
||||
<td key={column}
|
||||
className="overflow-hidden break-words border-b px-2 py-1
|
||||
align-top dark:border-slate-700"
|
||||
data-counts-copy={countsCellCopy (
|
||||
row,
|
||||
fields.find (field => field.key === column)!, params, locale)}>
|
||||
{cell (row, column)}</td>))}
|
||||
</tr>))}
|
||||
</tbody>
|
||||
</CountsSelectableTable>)}
|
||||
{(data.versions?.length ?? 0) === 0
|
||||
&& <p className="p-2">{label ('empty')}</p>}
|
||||
</div>)
|
||||
}}</SortableColumns>
|
||||
}
|
||||
|
||||
|
||||
export default CountsResults
|
||||
@@ -0,0 +1,17 @@
|
||||
import CountsResizeHandle from '@/components/counts/CountsResizeHandle'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
|
||||
const CountsRowNumber = ({ value, rowSpan }: { value?: number; rowSpan?: number }) => {
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
if (value != null) return <td data-counts-copy={String (value)}
|
||||
className="border-b px-2 py-1 text-right align-top tabular-nums dark:border-slate-700">
|
||||
{value}</td>
|
||||
return <th rowSpan={rowSpan} data-counts-heading={message ('rowNumberHeading').text} className="relative px-2 py-1">
|
||||
{label ('rowNumberHeading')}
|
||||
<CountsResizeHandle field="row_number" label={message ('rowNumberHeading')}/>
|
||||
</th>
|
||||
}
|
||||
|
||||
|
||||
export default CountsRowNumber
|
||||
@@ -0,0 +1,631 @@
|
||||
import { useCallback, useEffect, useId, useLayoutEffect, useMemo, useRef, useState } from 'react'
|
||||
import { createPortal } from 'react-dom'
|
||||
|
||||
import CountsCellMenu from '@/components/counts/CountsCellMenu'
|
||||
import { countsClipboard, writeCountsClipboard } from '@/lib/countsClipboard'
|
||||
import { CountsTableScope, useCountsTableLayout } from '@/lib/countsTableLayout'
|
||||
import { cn } from '@/lib/utils'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import type { ClipboardEvent, ReactNode } from 'react'
|
||||
|
||||
import type { CountsClipboardTable } from '@/lib/countsClipboard'
|
||||
import type { CountsMenuAction } from '@/components/counts/CountsCellMenu'
|
||||
|
||||
type Point = { row: number; column: number }
|
||||
type Selection = { identity: string; anchor: Point; end: Point }
|
||||
type TextMode = { identity: string; cell: HTMLTableCellElement; nodes: Node[] }
|
||||
type Caret = { cell: HTMLTableCellElement; left: number; top: number; height: number }
|
||||
const interactive = 'a,button,input,select,textarea,summary,[contenteditable],[data-counts-resize]'
|
||||
|
||||
|
||||
const cellGrid = (table: HTMLTableElement): HTMLTableCellElement[][] => {
|
||||
const grid: HTMLTableCellElement[][] = []
|
||||
for (const [row, tr] of Array.from (table.tBodies[0]?.rows ?? []).entries ())
|
||||
{
|
||||
grid[row] ??= []
|
||||
let column = 0
|
||||
for (const cell of Array.from (tr.cells))
|
||||
{
|
||||
while (grid[row][column] != null) ++column
|
||||
for (let y = 0; y < cell.rowSpan; ++y)
|
||||
{
|
||||
grid[row + y] ??= []
|
||||
for (let x = 0; x < cell.colSpan; ++x) grid[row + y][column + x] = cell
|
||||
}
|
||||
column += cell.colSpan
|
||||
}
|
||||
}
|
||||
return grid
|
||||
}
|
||||
|
||||
|
||||
const cellAt = (table: HTMLTableElement, point: Point) =>
|
||||
cellGrid (table)[point.row]?.[point.column]
|
||||
|
||||
|
||||
const scrollToCell = (table: HTMLTableElement, point: Point, previous: Point) => {
|
||||
const cell = cellAt (table, point)
|
||||
const container = table.parentElement?.closest<HTMLElement> (
|
||||
'[data-counts-scroll],[data-counts-results]')
|
||||
if (cell == null || container == null) return
|
||||
const viewport = container.getBoundingClientRect ()
|
||||
const top = viewport.top + container.clientTop
|
||||
const left = viewport.left + container.clientLeft
|
||||
const bottom = top + container.clientHeight
|
||||
const right = left + container.clientWidth
|
||||
const header = table.tHead?.getBoundingClientRect ()
|
||||
const visibleTop = header == null ? top : Math.max (top, Math.min (bottom, header.bottom))
|
||||
const rect = cell.getBoundingClientRect ()
|
||||
const lastRow = (table.tBodies[0]?.rows.length ?? 1) - 1
|
||||
const verticalEdge = (point.row === 0 || point.row < previous.row) ? 'start'
|
||||
: ((point.row === lastRow || point.row > previous.row) ? 'end'
|
||||
: (Math.abs (rect.top - visibleTop) <= Math.abs (rect.bottom - bottom) ? 'start' : 'end'))
|
||||
const offset = (
|
||||
start: number, end: number, minimum: number, maximum: number,
|
||||
edge?: 'start' | 'end',
|
||||
) => {
|
||||
if (maximum <= minimum) return 0
|
||||
const oversized = end - start > maximum - minimum
|
||||
// Vertical merged cells follow the destination and movement direction.
|
||||
// Horizontal scrolling retains its existing nearest-edge behaviour.
|
||||
if (oversized && edge != null) return edge === 'start' ? start - minimum : end - maximum
|
||||
if (start < minimum && end > maximum) return 0
|
||||
if (start < minimum) return oversized ? end - maximum : start - minimum
|
||||
if (end > maximum) return oversized ? start - minimum : end - maximum
|
||||
return 0
|
||||
}
|
||||
const vertical = offset (rect.top, rect.bottom, visibleTop, bottom, verticalEdge)
|
||||
const horizontal = offset (rect.left, rect.right, left, right)
|
||||
if (vertical !== 0) container.scrollTop += vertical
|
||||
if (horizontal !== 0) container.scrollLeft += horizontal
|
||||
}
|
||||
|
||||
|
||||
const CountsSelectableTable = (
|
||||
{ family, columns, identity, children, className, name, copyTable }: {
|
||||
family: string
|
||||
columns: string[]
|
||||
identity: string
|
||||
children: ReactNode
|
||||
className?: string
|
||||
name?: { text: string; language: string }
|
||||
copyTable?: () => CountsClipboardTable },
|
||||
) => {
|
||||
const id = useId ()
|
||||
const root = useRef<HTMLTableElement> (null)
|
||||
const [textMode, setTextMode] = useState<TextMode | null> (null)
|
||||
const textCell = textMode?.identity === identity ? textMode.cell : null
|
||||
const [caret, setCaret] = useState<Caret | null> (null)
|
||||
const mousePress = useRef<{ x: number; y: number } | null> (null)
|
||||
const dragging = useRef (false)
|
||||
const touchArmed = useRef (false)
|
||||
const touchPress = useRef<{ x: number; y: number
|
||||
timer: ReturnType<typeof setTimeout> } | null> (null)
|
||||
const [menu, setMenu] = useState<{ x: number; y: number; point: Point
|
||||
identity: string } | null> (null)
|
||||
const suppressClick = useRef (false)
|
||||
const [stored, setStored] = useState<Selection | null> (null)
|
||||
const selection = stored?.identity === identity ? stored : null
|
||||
const layout = useCountsTableLayout ()
|
||||
const { label, message } = useUITranslation ('counts')
|
||||
const clearTextSelection = useCallback (() => {
|
||||
setTextMode (null)
|
||||
setCaret (null)
|
||||
}, [])
|
||||
useLayoutEffect (() => {
|
||||
if (textMode == null) return
|
||||
const { cell, nodes } = textMode
|
||||
const table = root.current
|
||||
if (table == null || textMode.identity !== identity || !table.contains (cell))
|
||||
{
|
||||
setTextMode (null)
|
||||
return
|
||||
}
|
||||
cell.dataset.countsText = ''
|
||||
table.focus ({ preventScroll: true })
|
||||
const range = document.createRange ()
|
||||
range.setStart (nodes[0], 0)
|
||||
const last = nodes[nodes.length - 1]
|
||||
range.setEnd (last, last.textContent?.length ?? 0)
|
||||
const text = window.getSelection ()
|
||||
text?.removeAllRanges ()
|
||||
text?.addRange (range)
|
||||
return () => {
|
||||
delete cell.dataset.countsText
|
||||
const text = window.getSelection ()
|
||||
if (text != null && cell.contains (text.anchorNode)) text.removeAllRanges ()
|
||||
}
|
||||
}, [identity, textMode])
|
||||
useLayoutEffect (() => {
|
||||
if (textCell == null) return
|
||||
const updateCaret = () => {
|
||||
const text = window.getSelection ()
|
||||
if (!document.hasFocus () || text == null || !text.isCollapsed || text.rangeCount === 0
|
||||
|| !textCell.contains (text.focusNode))
|
||||
{
|
||||
setCaret (null)
|
||||
return
|
||||
}
|
||||
const range = text.getRangeAt (0).cloneRange ()
|
||||
const rect = range.getBoundingClientRect ()
|
||||
const node = text.focusNode
|
||||
const offset = text.focusOffset
|
||||
const probe = range.cloneRange ()
|
||||
if (rect.height === 0 && node?.nodeType === Node.TEXT_NODE && node.textContent)
|
||||
{
|
||||
if (offset < node.textContent.length) probe.setEnd (node, offset + 1)
|
||||
else probe.setStart (node, offset - 1)
|
||||
}
|
||||
const bounds = rect.height > 0 ? rect : probe.getBoundingClientRect ()
|
||||
if (bounds.height === 0)
|
||||
{
|
||||
setCaret (null)
|
||||
return
|
||||
}
|
||||
const cell = textCell.getBoundingClientRect ()
|
||||
const left = (rect.height === 0 && offset === node?.textContent?.length)
|
||||
? bounds.right : bounds.left
|
||||
setCaret ({ cell: textCell,
|
||||
left: left - cell.left - textCell.clientLeft + textCell.scrollLeft,
|
||||
top: bounds.top - cell.top - textCell.clientTop + textCell.scrollTop,
|
||||
height: bounds.height })
|
||||
}
|
||||
updateCaret ()
|
||||
document.addEventListener ('selectionchange', updateCaret)
|
||||
document.addEventListener ('scroll', updateCaret, true)
|
||||
window.addEventListener ('resize', updateCaret)
|
||||
window.addEventListener ('focus', updateCaret)
|
||||
window.addEventListener ('blur', updateCaret)
|
||||
return () => {
|
||||
document.removeEventListener ('selectionchange', updateCaret)
|
||||
document.removeEventListener ('scroll', updateCaret, true)
|
||||
window.removeEventListener ('resize', updateCaret)
|
||||
window.removeEventListener ('focus', updateCaret)
|
||||
window.removeEventListener ('blur', updateCaret)
|
||||
}
|
||||
}, [textCell])
|
||||
useEffect (() => {
|
||||
if (textCell == null) return
|
||||
const outside = (event: PointerEvent) => {
|
||||
if (!textCell.contains (event.target as Node)) clearTextSelection ()
|
||||
}
|
||||
document.addEventListener ('pointerdown', outside, true)
|
||||
return () => document.removeEventListener ('pointerdown', outside, true)
|
||||
}, [clearTextSelection, textCell])
|
||||
const bounds = useMemo (() => selection == null ? null : {
|
||||
top: Math.min (selection.anchor.row, selection.end.row),
|
||||
bottom: Math.max (selection.anchor.row, selection.end.row),
|
||||
left: Math.min (selection.anchor.column, selection.end.column),
|
||||
right: Math.max (selection.anchor.column, selection.end.column) }, [selection])
|
||||
const finish = () => {
|
||||
mousePress.current = null
|
||||
dragging.current = false
|
||||
touchArmed.current = false
|
||||
if (touchPress.current != null) clearTimeout (touchPress.current.timer)
|
||||
touchPress.current = null
|
||||
}
|
||||
useEffect (() => {
|
||||
const table = root.current
|
||||
const preventPan = (event: TouchEvent) => {
|
||||
if (touchArmed.current) event.preventDefault ()
|
||||
}
|
||||
table?.addEventListener ('touchmove', preventPan, { passive: false })
|
||||
return () => {
|
||||
table?.removeEventListener ('touchmove', preventPan)
|
||||
if (touchPress.current != null) clearTimeout (touchPress.current.timer)
|
||||
}
|
||||
}, [])
|
||||
useLayoutEffect (() => {
|
||||
if (root.current == null) return
|
||||
if (textCell != null && !root.current.contains (textCell)) setTextMode (null)
|
||||
const grid = cellGrid (root.current)
|
||||
const assigned = new Set<HTMLTableCellElement> ()
|
||||
for (const [row, cells] of grid.entries ())
|
||||
{
|
||||
for (const [column, cell] of cells.entries ())
|
||||
{
|
||||
if (assigned.has (cell)) continue
|
||||
assigned.add (cell)
|
||||
const selected = bounds != null && row <= bounds.bottom
|
||||
&& row + cell.rowSpan - 1 >= bounds.top && column <= bounds.right
|
||||
&& column + cell.colSpan - 1 >= bounds.left
|
||||
cell.id = `${id}-${row}-${column}`
|
||||
if (cell.getAttribute ('role') !== 'rowheader') cell.setAttribute ('role', 'gridcell')
|
||||
cell.setAttribute ('aria-selected', String (selected))
|
||||
if (selected && cell !== textCell) cell.dataset.countsSelected = ''
|
||||
else delete cell.dataset.countsSelected
|
||||
}
|
||||
}
|
||||
if (bounds != null)
|
||||
root.current.setAttribute ('aria-activedescendant', grid[bounds.bottom]?.[bounds.right]?.id ?? '')
|
||||
else root.current.removeAttribute ('aria-activedescendant')
|
||||
}, [bounds, children, id, textCell])
|
||||
const pointFor = (element: Element | null, y?: number): Point | null => {
|
||||
const table = root.current
|
||||
if (table == null || element == null || !table.contains (element)) return null
|
||||
let cell = element.closest ('td')
|
||||
while (cell != null && cell.closest ('table') !== table)
|
||||
cell = cell.closest ('table')?.parentElement?.closest ('td') ?? null
|
||||
if (cell == null || cell.parentElement?.parentElement !== table.tBodies[0]) return null
|
||||
const grid = cellGrid (table)
|
||||
const hitRow = y == null ? -1 : Array.from (table.tBodies[0].rows).findIndex (row => {
|
||||
const rect = row.getBoundingClientRect ()
|
||||
return y >= rect.top && y < rect.bottom
|
||||
})
|
||||
for (const [row, cells] of grid.entries ())
|
||||
{
|
||||
if (hitRow >= 0 && row !== hitRow) continue
|
||||
const column = cells.indexOf (cell as HTMLTableCellElement)
|
||||
if (column >= 0) return { row, column }
|
||||
}
|
||||
return null
|
||||
}
|
||||
const enterTextSelection = (point: Point) => {
|
||||
const table = root.current
|
||||
const cell = table == null ? null : cellAt (table, point)
|
||||
if (table == null || cell == null) return
|
||||
// Structured cells retain their independent inner grid instead of selecting its UI.
|
||||
const nested = cell.querySelector ('table')
|
||||
if (nested != null)
|
||||
{
|
||||
nested.focus ({ preventScroll: true })
|
||||
return
|
||||
}
|
||||
const walker = document.createTreeWalker (cell, NodeFilter.SHOW_TEXT, {
|
||||
acceptNode: node => {
|
||||
const parent = node.parentElement
|
||||
const auxiliaryFlg = parent?.closest (
|
||||
'[hidden],[aria-hidden="true"],.sr-only,svg,[data-counts-resize]') != null
|
||||
return (parent?.closest ('table') === table && node.textContent?.trim ()
|
||||
&& !auxiliaryFlg)
|
||||
? NodeFilter.FILTER_ACCEPT : NodeFilter.FILTER_REJECT
|
||||
} })
|
||||
const nodes: Node[] = []
|
||||
for (let node = walker.nextNode (); node != null; node = walker.nextNode ())
|
||||
nodes.push (node)
|
||||
if (nodes.length === 0) return
|
||||
finish ()
|
||||
clearTextSelection ()
|
||||
setMenu (null)
|
||||
setTextMode ({ identity, cell, nodes })
|
||||
}
|
||||
const closeMenu = useCallback (() => {
|
||||
setMenu (null)
|
||||
root.current?.focus ({ preventScroll: true })
|
||||
}, [])
|
||||
const openMenu = (point: Point, x: number, y: number) => {
|
||||
const inside = bounds != null && point.row >= bounds.top && point.row <= bounds.bottom
|
||||
&& point.column >= bounds.left && point.column <= bounds.right
|
||||
if (inside) setStored (selection)
|
||||
else setStored ({ identity, anchor: point, end: point })
|
||||
setMenu ({ point, x, y, identity })
|
||||
}
|
||||
const clipboard = (headers = false) => {
|
||||
if (bounds == null || root.current == null) return null
|
||||
if (copyTable != null)
|
||||
{
|
||||
const table = copyTable ()
|
||||
return countsClipboard ({
|
||||
columns: table.columns.slice (bounds.left, bounds.right + 1),
|
||||
headerRows: table.headerRows?.map (row => row.slice (bounds.left, bounds.right + 1)),
|
||||
rows: table.rows.slice (bounds.top, bounds.bottom + 1)
|
||||
.map (row => row.slice (bounds.left, bounds.right + 1)) }, headers)
|
||||
}
|
||||
const rows: string[][] = []
|
||||
for (let row = bounds.top; row <= bounds.bottom; ++row)
|
||||
{
|
||||
const values: string[] = []
|
||||
for (let column = bounds.left; column <= bounds.right; ++column)
|
||||
{
|
||||
const cell = cellAt (root.current, { row, column })
|
||||
values.push (cell?.dataset.countsCopy ?? cell?.innerText ?? '')
|
||||
}
|
||||
rows.push (values)
|
||||
}
|
||||
if (headers)
|
||||
{
|
||||
const heading = root.current.tHead?.rows[0]
|
||||
rows.unshift (Array.from ({ length: bounds.right - bounds.left + 1 }, (entry, index) => {
|
||||
void entry
|
||||
const cell = heading?.cells[bounds.left + index]
|
||||
return cell?.dataset.countsHeading ?? cell?.textContent ?? ''
|
||||
}))
|
||||
}
|
||||
return countsClipboard (rows, headers)
|
||||
}
|
||||
const menuAction = (action: CountsMenuAction) => {
|
||||
if (menu == null || root.current == null) return
|
||||
if (action === 'copy' || action === 'copyHeaders')
|
||||
{
|
||||
const payload = clipboard (action === 'copyHeaders')
|
||||
if (payload != null) void writeCountsClipboard (payload).catch (error => {
|
||||
console.error (error)
|
||||
})
|
||||
}
|
||||
else
|
||||
{
|
||||
const lastRow = (root.current.tBodies[0]?.rows.length ?? 1) - 1
|
||||
const lastColumn = columns.length - 1
|
||||
const anchor = {
|
||||
row: action === 'row' ? menu.point.row : 0,
|
||||
column: action === 'column' ? menu.point.column : 0 }
|
||||
const end = {
|
||||
row: action === 'row' ? menu.point.row : lastRow,
|
||||
column: action === 'column' ? menu.point.column : lastColumn }
|
||||
setStored ({ identity, anchor, end })
|
||||
}
|
||||
closeMenu ()
|
||||
}
|
||||
const copy = (event: ClipboardEvent<HTMLTableElement>) => {
|
||||
const text = window.getSelection ()
|
||||
if (textCell != null && text != null && !text.isCollapsed
|
||||
&& textCell.contains (text.anchorNode)
|
||||
&& textCell.contains (text.focusNode))
|
||||
{
|
||||
event.stopPropagation ()
|
||||
return
|
||||
}
|
||||
if (document.activeElement !== root.current) return
|
||||
const payload = clipboard ()
|
||||
if (payload == null) return
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
event.clipboardData.setData ('text/plain', payload.text)
|
||||
try
|
||||
{
|
||||
event.clipboardData.setData ('text/html', payload.html)
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Plain TSV remains available when the browser disallows HTML clipboard data.
|
||||
}
|
||||
}
|
||||
const totalWidth = columns.reduce ((sum, key) => sum + layout.width (family, key), 0)
|
||||
return (
|
||||
<CountsTableScope.Provider value={family}>
|
||||
<table ref={root} role="grid" tabIndex={0} aria-multiselectable="true"
|
||||
aria-label={name?.text ?? message ('selection.table').text}
|
||||
aria-describedby={`${id}-hint`}
|
||||
|
||||
style={{ tableLayout: 'fixed', minWidth: totalWidth, width: totalWidth }}
|
||||
className={cn ('counts-grid select-none border-separate border-spacing-0 outline-none',
|
||||
'focus-visible:outline focus-visible:outline-2 focus-visible:outline-blue-500',
|
||||
'dark:focus-visible:outline-blue-400',
|
||||
'[&>tbody>tr>td[data-counts-selected]]:bg-blue-100',
|
||||
'[&>tbody>tr>td[data-counts-selected]]:text-slate-900',
|
||||
'dark:[&>tbody>tr>td[data-counts-selected]]:bg-blue-900',
|
||||
'dark:[&>tbody>tr>td[data-counts-selected]]:text-slate-100',
|
||||
'[&>tbody>tr>td[data-counts-selected]]:outline',
|
||||
'[&>tbody>tr>td[data-counts-selected]]:outline-blue-500',
|
||||
'[&>tbody>tr>td[data-counts-selected]]:outline-offset-[-1px]', className)}
|
||||
onCopy={copy}
|
||||
onDoubleClick={event => {
|
||||
const element = event.target as Element
|
||||
if (element.closest ('table') !== event.currentTarget
|
||||
|| element.closest (interactive) != null) return
|
||||
if (textCell?.contains (element))
|
||||
{
|
||||
event.stopPropagation ()
|
||||
return
|
||||
}
|
||||
const point = pointFor (element, event.clientY)
|
||||
if (point == null) return
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
enterTextSelection (point)
|
||||
}}
|
||||
onClickCapture={event => {
|
||||
if ((event.target as Element).closest ('table') !== event.currentTarget
|
||||
|| !suppressClick.current) return
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
suppressClick.current = false
|
||||
}}
|
||||
onPointerDown={event => {
|
||||
const element = event.target as Element
|
||||
if (element.closest ('table') !== event.currentTarget) return
|
||||
suppressClick.current = false
|
||||
if (textCell != null && textCell.contains (element))
|
||||
{
|
||||
event.stopPropagation ()
|
||||
return
|
||||
}
|
||||
clearTextSelection ()
|
||||
if (element.closest (interactive) != null || event.button !== 0) return
|
||||
if (element instanceof HTMLElement && element.hasAttribute ('data-counts-scroll'))
|
||||
{
|
||||
const rect = element.getBoundingClientRect ()
|
||||
if (event.clientX >= rect.left + element.clientLeft + element.clientWidth
|
||||
|| event.clientY >= rect.top + element.clientTop + element.clientHeight) return
|
||||
}
|
||||
const point = pointFor (element, event.clientY)
|
||||
if (point == null) return
|
||||
event.stopPropagation ()
|
||||
setMenu (null)
|
||||
event.currentTarget.focus ({ preventScroll: true })
|
||||
setStored ({ identity, anchor: event.shiftKey && selection != null
|
||||
? selection.anchor : point, end: point })
|
||||
finish ()
|
||||
if (event.pointerType === 'mouse')
|
||||
{
|
||||
mousePress.current = { x: event.clientX, y: event.clientY }
|
||||
}
|
||||
else
|
||||
{
|
||||
const table = event.currentTarget
|
||||
const pointerId = event.pointerId
|
||||
touchPress.current = { x: event.clientX, y: event.clientY,
|
||||
timer: setTimeout (() => {
|
||||
suppressClick.current = true
|
||||
openMenu (point, event.clientX, event.clientY)
|
||||
dragging.current = true
|
||||
touchArmed.current = true
|
||||
table.setPointerCapture (pointerId)
|
||||
}, 450) }
|
||||
}
|
||||
}}
|
||||
onPointerMove={event => {
|
||||
if (mousePress.current != null)
|
||||
{
|
||||
if (event.buttons === 0) finish ()
|
||||
else if (!dragging.current && Math.hypot (
|
||||
event.clientX - mousePress.current.x, event.clientY - mousePress.current.y) > 5)
|
||||
{
|
||||
dragging.current = true
|
||||
suppressClick.current = true
|
||||
event.currentTarget.setPointerCapture (event.pointerId)
|
||||
}
|
||||
}
|
||||
if (touchPress.current != null && !touchArmed.current && Math.hypot (
|
||||
event.clientX - touchPress.current.x, event.clientY - touchPress.current.y) > 8)
|
||||
finish ()
|
||||
if (!dragging.current || event.buttons === 0) return
|
||||
event.preventDefault ()
|
||||
const point = pointFor (document.elementFromPoint (event.clientX, event.clientY), event.clientY)
|
||||
if (point == null) return
|
||||
if (touchArmed.current && touchPress.current != null
|
||||
&& Math.hypot (event.clientX - touchPress.current.x,
|
||||
event.clientY - touchPress.current.y) <= 8) return
|
||||
setMenu (null)
|
||||
event.stopPropagation ()
|
||||
setStored (previous => previous?.identity !== identity ? previous : {
|
||||
...previous, end: point })
|
||||
}}
|
||||
onPointerUp={finish} onPointerCancel={finish}
|
||||
onLostPointerCapture={event => {
|
||||
if (event.target === event.currentTarget) finish ()
|
||||
}}
|
||||
onContextMenu={event => {
|
||||
const element = event.target as Element
|
||||
if (element.closest ('table') !== event.currentTarget) return
|
||||
if (textCell != null && textCell.contains (element))
|
||||
{
|
||||
event.stopPropagation ()
|
||||
return
|
||||
}
|
||||
const point = pointFor (element, event.clientY)
|
||||
if (point == null || element.closest ('[data-counts-resize]') != null) return
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
openMenu (point, event.clientX, event.clientY)
|
||||
}}
|
||||
onKeyDown={event => {
|
||||
if (event.target !== event.currentTarget) return
|
||||
if (textCell == null
|
||||
&& (event.key === 'ContextMenu' || (event.shiftKey && event.key === 'F10')))
|
||||
{
|
||||
const point = selection?.end ?? { row: 0, column: 0 }
|
||||
const rect = cellAt (event.currentTarget, point)?.getBoundingClientRect ()
|
||||
if (rect != null)
|
||||
{
|
||||
event.preventDefault ()
|
||||
openMenu (point, rect.left, rect.bottom)
|
||||
}
|
||||
return
|
||||
}
|
||||
if (event.key === 'Escape' || (event.key === 'Enter' && textCell != null))
|
||||
{
|
||||
const textModeFlg = textCell != null
|
||||
clearTextSelection ()
|
||||
if (textModeFlg)
|
||||
{
|
||||
window.getSelection ()?.removeAllRanges ()
|
||||
event.currentTarget.focus ({ preventScroll: true })
|
||||
}
|
||||
else setStored (null)
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
return
|
||||
}
|
||||
if (event.key === 'Enter' && textCell == null && selection != null)
|
||||
{
|
||||
event.preventDefault ()
|
||||
enterTextSelection (selection.end)
|
||||
return
|
||||
}
|
||||
if (textCell != null)
|
||||
{
|
||||
const boundaryFlg = event.key === 'Home' || event.key === 'End'
|
||||
if (!boundaryFlg
|
||||
&& !['ArrowUp', 'ArrowDown', 'ArrowLeft', 'ArrowRight'].includes (event.key)) return
|
||||
const text = window.getSelection ()
|
||||
if (text == null) return
|
||||
if (boundaryFlg && (event.ctrlKey || event.metaKey))
|
||||
{
|
||||
const nodes = textMode?.nodes.filter (node => textCell.contains (node)) ?? []
|
||||
const edge = event.key === 'Home' ? nodes[0] : nodes[nodes.length - 1]
|
||||
if (edge == null) return
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
const end = event.key === 'Home' ? 0 : edge.textContent?.length ?? 0
|
||||
if (event.shiftKey && text.anchorNode != null && textCell.contains (text.anchorNode))
|
||||
text.setBaseAndExtent (text.anchorNode, text.anchorOffset, edge, end)
|
||||
else text.setPosition (edge, end)
|
||||
return
|
||||
}
|
||||
if (typeof text.modify !== 'function') return
|
||||
event.preventDefault ()
|
||||
event.stopPropagation ()
|
||||
const anchor = text.anchorNode
|
||||
const offset = text.anchorOffset
|
||||
const backward = ['ArrowLeft', 'ArrowUp', 'Home'].includes (event.key)
|
||||
const vertical = event.key === 'ArrowUp' || event.key === 'ArrowDown'
|
||||
const direction = (vertical || boundaryFlg) ? (backward ? 'backward' : 'forward')
|
||||
: (backward ? 'left' : 'right')
|
||||
const grain = boundaryFlg ? 'lineboundary' : (vertical ? 'line'
|
||||
: ((event.ctrlKey || event.metaKey) ? 'word' : 'character'))
|
||||
text.modify (event.shiftKey ? 'extend' : 'move', direction, grain)
|
||||
if (!textCell.contains (text.focusNode))
|
||||
{
|
||||
const nodes = textMode?.nodes.filter (node => textCell.contains (node)) ?? []
|
||||
const edge = backward ? nodes[0] : nodes[nodes.length - 1]
|
||||
if (edge != null)
|
||||
{
|
||||
const end = backward ? 0 : edge.textContent?.length ?? 0
|
||||
if (event.shiftKey && anchor != null && textCell.contains (anchor))
|
||||
text.setBaseAndExtent (anchor, offset, edge, end)
|
||||
else text.setPosition (edge, end)
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
const boundaryFlg = event.key === 'Home' || event.key === 'End'
|
||||
if (!boundaryFlg
|
||||
&& !['ArrowUp', 'ArrowDown', 'ArrowLeft', 'ArrowRight'].includes (event.key)) return
|
||||
const rows = event.currentTarget.tBodies[0]?.rows
|
||||
if (rows == null || rows.length === 0) return
|
||||
event.preventDefault ()
|
||||
const previous = selection?.end ?? { row: 0, column: 0 }
|
||||
const key = (selection == null && !boundaryFlg) ? '' : event.key
|
||||
const first = event.key === 'Home'
|
||||
const vertical = boundaryFlg && (event.ctrlKey || event.metaKey)
|
||||
const point = {
|
||||
row: vertical ? (first ? 0 : rows.length - 1)
|
||||
: Math.max (0, Math.min (rows.length - 1, previous.row
|
||||
+ (key === 'ArrowDown' ? 1 : (key === 'ArrowUp' ? -1 : 0)))),
|
||||
column: (boundaryFlg && !vertical) ? (first ? 0 : columns.length - 1)
|
||||
: Math.max (0, Math.min (columns.length - 1, previous.column
|
||||
+ (key === 'ArrowRight' ? 1 : (key === 'ArrowLeft' ? -1 : 0)))) }
|
||||
setStored ({ identity, anchor: event.shiftKey && selection != null
|
||||
? selection.anchor : point, end: point })
|
||||
scrollToCell (event.currentTarget, point, previous)
|
||||
}}>
|
||||
<caption className="sr-only">
|
||||
<span id={`${id}-hint`} className="sr-only">{label ('selection.hint')}</span>
|
||||
<span role="status">{textCell != null && label ('selection.textMode')}</span>
|
||||
</caption>
|
||||
<colgroup>{columns.map (key => <col key={key}
|
||||
style={{ width: layout.width (family, key) }}/>)}</colgroup>
|
||||
{children}
|
||||
</table>
|
||||
{textCell != null && caret != null && caret.cell === textCell && createPortal (
|
||||
<span aria-hidden="true" className="counts-text-caret" style={{
|
||||
left: caret.left, top: caret.top, height: caret.height }}/>, textCell)}
|
||||
{menu?.identity === identity && <CountsCellMenu x={menu.x} y={menu.y}
|
||||
act={menuAction} close={closeMenu}/>}
|
||||
</CountsTableScope.Provider>)
|
||||
}
|
||||
|
||||
|
||||
export default CountsSelectableTable
|
||||
@@ -0,0 +1,145 @@
|
||||
import { useContext, useEffect, useRef } from 'react'
|
||||
|
||||
import SortableItem from '@/components/common/SortableItem'
|
||||
import CountsResizeHandle from '@/components/counts/CountsResizeHandle'
|
||||
import { Button } from '@/components/ui/button'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
import { CountsTableScope, useCountsTableLayout } from '@/lib/countsTableLayout'
|
||||
|
||||
import type { CountSort } from '@/lib/counts'
|
||||
|
||||
|
||||
const CountsSortHeader = (
|
||||
{ field, label, sorts, change, structured = false, rowSpan, colSpan, resizable = true, dragId, totalKind }: {
|
||||
field: string
|
||||
label: { text: string; language: string }
|
||||
sorts: CountSort[]
|
||||
change: (key: string, additive: boolean) => void
|
||||
structured?: boolean
|
||||
rowSpan?: number; colSpan?: number; resizable?: boolean
|
||||
dragId?: string; totalKind?: string },
|
||||
) => {
|
||||
const { message } = useUITranslation ('counts')
|
||||
const family = useContext (CountsTableScope)
|
||||
const layout = useCountsTableLayout ()
|
||||
const heading = useRef<HTMLDivElement> (null)
|
||||
useEffect (() => {
|
||||
if (!resizable || colSpan != null && colSpan > 1) return
|
||||
const node = heading.current
|
||||
const text = node?.querySelector<HTMLElement> ('[data-counts-label]')
|
||||
const button = text?.closest ('button')
|
||||
if (node == null || text == null || button == null) return
|
||||
const canvas = document.createElement ('canvas')
|
||||
const context = canvas.getContext ('2d')
|
||||
if (context == null) return
|
||||
const measure = () => {
|
||||
context.font = getComputedStyle (text).font
|
||||
const padding = (element: Element) => {
|
||||
const style = getComputedStyle (element)
|
||||
return parseFloat (style.paddingLeft) + parseFloat (style.paddingRight)
|
||||
+ parseFloat (style.borderLeftWidth) + parseFloat (style.borderRightWidth)
|
||||
}
|
||||
const handles = Array.from (node.children).filter (child => child !== button)
|
||||
.reduce ((sum, child) => sum + Math.max (
|
||||
child.scrollWidth, child.getBoundingClientRect ().width), 0)
|
||||
const resize = node.parentElement?.querySelector ('[data-counts-resize]')
|
||||
const resizeWidth = resize?.getBoundingClientRect ().width ?? 0
|
||||
const indicator = button.querySelector ('[data-counts-sort-indicator]')
|
||||
const sortWidth = indicator?.getBoundingClientRect ().width ?? 0
|
||||
const gap = parseFloat (getComputedStyle (button).columnGap) || 0
|
||||
layout.fit (family, field, context.measureText (label.text).width
|
||||
+ handles + padding (button) + padding (node.parentElement!)
|
||||
+ sortWidth + gap + resizeWidth)
|
||||
}
|
||||
measure ()
|
||||
let active = true
|
||||
void document.fonts.ready.then (() => { if (active) measure () })
|
||||
return () => { active = false }
|
||||
}, [family, field, label.text, dragId, resizable, colSpan, layout])
|
||||
const press = useRef<{ x: number; y: number
|
||||
timer: ReturnType<typeof setTimeout> } | null> (null)
|
||||
const suppressClick = useRef (false)
|
||||
const cancel = () => {
|
||||
if (press.current != null) clearTimeout (press.current.timer)
|
||||
press.current = null
|
||||
}
|
||||
useEffect (() => () => {
|
||||
if (press.current != null) clearTimeout (press.current.timer)
|
||||
}, [])
|
||||
const priority = sorts.findIndex (sort => sort.key === field)
|
||||
const direction = sorts[priority]?.direction
|
||||
const state = priority < 0 ? 'none' : (direction === 'asc' ? 'ascending' : 'descending')
|
||||
const title = message (structured ? 'sort.structured' : 'sort.action')
|
||||
const aria = message (priority < 0 ? 'sort.unsortedLabel'
|
||||
: (sorts.length === 1 ? 'sort.singleLabel' : 'sort.label'), {
|
||||
column: label.text, priority: priority + 1,
|
||||
direction: message (`sort.${state}`).text })
|
||||
|
||||
const render = (shell?: Parameters<NonNullable<
|
||||
Parameters<typeof SortableItem>[0]['renderShell']>>[0]) => (
|
||||
<th ref={shell?.ref} style={shell?.style} data-counts-total={totalKind}
|
||||
data-counts-heading={label.text} rowSpan={rowSpan} colSpan={colSpan}
|
||||
className="relative overflow-hidden whitespace-nowrap px-1 py-1"
|
||||
aria-sort={(sorts.length > 1 && priority >= 0) ? 'other' : state}>
|
||||
<div ref={heading} className="flex min-w-0 items-center">
|
||||
{shell?.handle}
|
||||
<Button type="button" variant="ghost" size="sm"
|
||||
className="h-auto w-full max-w-full justify-start select-none px-1 py-0 pr-3"
|
||||
lang={aria.language} aria-label={aria.text} title={title.text}
|
||||
onPointerDown={event => {
|
||||
if (!event.isPrimary)
|
||||
{
|
||||
cancel ()
|
||||
return
|
||||
}
|
||||
suppressClick.current = false
|
||||
if (event.pointerType !== 'touch' && event.pointerType !== 'pen')
|
||||
return
|
||||
cancel ()
|
||||
const x = event.clientX
|
||||
const y = event.clientY
|
||||
press.current = { x, y, timer: setTimeout (() => {
|
||||
press.current = null
|
||||
suppressClick.current = true
|
||||
change (field, true)
|
||||
}, 500) }
|
||||
}}
|
||||
onPointerMove={event => {
|
||||
if (press.current != null && Math.hypot (
|
||||
event.clientX - press.current.x, event.clientY - press.current.y) > 8)
|
||||
cancel ()
|
||||
}}
|
||||
onPointerUp={cancel} onPointerCancel={cancel} onPointerLeave={cancel}
|
||||
onContextMenu={event => {
|
||||
if (press.current != null || suppressClick.current)
|
||||
event.preventDefault ()
|
||||
}}
|
||||
onKeyDown={event => {
|
||||
if (event.key !== 'Enter' && event.key !== ' ') return
|
||||
event.preventDefault ()
|
||||
suppressClick.current = false
|
||||
change (field, event.ctrlKey || event.metaKey)
|
||||
}}
|
||||
onClick={event => {
|
||||
if (suppressClick.current)
|
||||
{
|
||||
suppressClick.current = false
|
||||
event.preventDefault ()
|
||||
return
|
||||
}
|
||||
change (field, event.ctrlKey || event.metaKey)
|
||||
}}>
|
||||
<span data-counts-label className="truncate" lang={label.language}>{label.text}</span>
|
||||
<span aria-hidden data-counts-sort-indicator className="inline-block w-10 shrink-0">
|
||||
{priority >= 0 && <>{direction === 'asc' ? '▲' : '▼'}
|
||||
{sorts.length > 1 && priority + 1}</>}</span>
|
||||
</Button>
|
||||
</div>
|
||||
{resizable && <CountsResizeHandle field={field} label={label}/>}
|
||||
</th>)
|
||||
return dragId == null ? render () : <SortableItem id={dragId} name={message ('reorder', { column: label.text })}
|
||||
handleId={`column-${dragId}`} moveGroup={() => { }} renderShell={render}>{null}</SortableItem>
|
||||
}
|
||||
|
||||
|
||||
export default CountsSortHeader
|
||||
@@ -0,0 +1,312 @@
|
||||
import SortableColumns from '@/components/common/SortableColumns'
|
||||
import { countsSummaryValue } from '@/lib/countsDisplay'
|
||||
import PrefetchLink from '@/components/PrefetchLink'
|
||||
import CountsRowNumber from '@/components/counts/CountsRowNumber'
|
||||
import CountsSelectableTable from '@/components/counts/CountsSelectableTable'
|
||||
import CountsSortHeader from '@/components/counts/CountsSortHeader'
|
||||
import { changeCountSort, countsRequestParams, readCountSort, writeCountSort } from '@/lib/counts'
|
||||
import { countsAxisLabel, useCountsDefinition }
|
||||
from '@/lib/countsDefinitions'
|
||||
import { cn } from '@/lib/utils'
|
||||
import { useLocale } from '@/lib/useLocale'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import { queryRecordParams, setQueryRecords, readCountState, encodeCountState, setCountState } from '@/lib/queryState'
|
||||
|
||||
import type { CountsPivotHeader, CountsResource, CountsResult } from '@/lib/counts'
|
||||
import type { CountsAxis } from '@/lib/countsDefinitions'
|
||||
|
||||
|
||||
const CountsSummaryTable = (
|
||||
{ resource, data, params, pending, update, dragIds = {} }: {
|
||||
dragIds?: Record<string, string>
|
||||
resource: CountsResource; data: CountsResult; params: URLSearchParams
|
||||
pending: boolean; update: (key: string, value: string, extra?: Record<string, string>) => void },
|
||||
) => {
|
||||
const definition = useCountsDefinition ()
|
||||
const locale = useLocale ()
|
||||
const { message } = useUITranslation ('counts')
|
||||
const numbered = params.get ('row_numbers') === '1'
|
||||
const rows = data.rows ?? []
|
||||
const cols = data.cols ?? []
|
||||
const cells = data.cells ?? []
|
||||
const rowHeaders: CountsPivotHeader[] = data.rowHeaders ?? [...new Set (
|
||||
cells.map (cell => JSON.stringify (cell.values.slice (0, rows.length))))]
|
||||
.map (key => ({ values: JSON.parse (key), kind: 'data' }))
|
||||
const columnHeaders: CountsPivotHeader[] = data.columnHeaders
|
||||
?? (data.columnValues ?? [[]]).map (values => ({ values, kind: 'data' }))
|
||||
const sorts = readCountSort (params.get ('sort'))
|
||||
const timezone = params.get ('timezone') ?? 'UTC'
|
||||
const change = (key: string, additive: boolean) => update (
|
||||
'sort', writeCountSort (changeCountSort (sorts, key, additive)))
|
||||
const display = (value: string | number | null, axis: CountsAxis, index: number) =>
|
||||
countsSummaryValue (value, axis, index, data, locale, timezone)
|
||||
const aggregateLabel = (header: CountsPivotHeader) => message (
|
||||
header.labelKey ?? (header.kind === 'grand_total' ? (rows.length + cols.length === 1
|
||||
? 'singleTotal' : 'grandTotal') : 'subtotal')).text
|
||||
const leading = Math.max (rows.length, cols.length > 0 ? 1 : 0)
|
||||
const levels = Math.max (1, cols.length)
|
||||
const headingLevels = cols.length === 0 ? 1 : levels * 2
|
||||
const names = (header: CountsPivotHeader, axes: CountsAxis[], offset: number) =>
|
||||
header.values.map ((value, index) => display (value, axes[index], offset + index))
|
||||
const columnKey = (header: CountsPivotHeader) => `${header.kind}:${JSON.stringify (header.values)}`
|
||||
const measures = readCountState<Record<string, string | number | null>[]> (
|
||||
params, 'measures', [])
|
||||
const measureKey = (header: CountsPivotHeader) => {
|
||||
if (header.values.length === 0) return 'count'
|
||||
const ids = [...new Set (measures.map (entry => String (entry.key)))]
|
||||
const existing = ids.find (id => JSON.stringify (measures.filter (entry => entry.key === id)
|
||||
.sort ((a, b) => Number (a.axis) - Number (b.axis)).map (entry => entry.value))
|
||||
=== JSON.stringify (header.values))
|
||||
return existing ?? `m${Array.from ({ length: 100 }, (_, index) => index)
|
||||
.find (index => !ids.includes (`m${index}`)) ?? 99}`
|
||||
}
|
||||
const changeMeasure = (header: CountsPivotHeader, additive: boolean) => {
|
||||
const key = measureKey (header)
|
||||
const next = new URLSearchParams (params)
|
||||
const nextSort = changeCountSort (sorts, key, additive)
|
||||
const active = new Set (nextSort.map (sort => sort.key))
|
||||
const records = measures.filter (record => record.key !== key && active.has (String (record.key)))
|
||||
setQueryRecords (next, 'measures', [...records, ...(active.has (key)
|
||||
? header.values.map ((value, axis) => ({ key, axis, value })) : [])])
|
||||
update ('sort', writeCountSort (nextSort),
|
||||
queryRecordParams (next, 'measures'))
|
||||
}
|
||||
const keys = [...(numbered ? ['row_number'] : []),
|
||||
...Array.from ({ length: leading }, (_, index) => `a${index}`),
|
||||
...columnHeaders.map (columnKey)]
|
||||
const full = new Map (cells.map (cell => [JSON.stringify (cell.values), cell.count]))
|
||||
const totals = new Map ((data.pivotTotals ?? []).map (cell => [
|
||||
JSON.stringify ([cell.row, cell.column]), cell.count]))
|
||||
const totalLabels = new Map ((data.pivotTotals ?? []).map (cell => [
|
||||
JSON.stringify ([cell.row, cell.column]), cell.labelKey]))
|
||||
const count = (row: CountsPivotHeader, column: CountsPivotHeader) =>
|
||||
(row.values.length === rows.length && column.values.length === cols.length)
|
||||
? full.get (JSON.stringify ([...row.values, ...column.values])) ?? 0
|
||||
: totals.get (JSON.stringify ([row.values, column.values])) ?? 0
|
||||
const drill = (row: CountsPivotHeader, column: CountsPivotHeader) => {
|
||||
const next = new URLSearchParams (params)
|
||||
const existing = readCountState<Record<string, string | number | null>[]> (params, 'conditions', [])
|
||||
setCountState (next, 'conditions', encodeCountState ('conditions', [...existing,
|
||||
...row.values.map ((value, index) => ({ ...rows[index], operator: 'axis_eq', value })),
|
||||
...column.values.map ((value, index) => ({ ...cols[index], operator: 'axis_eq', value }))]))
|
||||
next.set ('mode', 'details')
|
||||
next.delete ('sort')
|
||||
next.delete ('page')
|
||||
return `/counts/${resource}?${next}`
|
||||
}
|
||||
let ordinal = (data.page - 1) * data.limit
|
||||
const rowLabels = rowHeaders.map (header => {
|
||||
const labels = names (header, rows, 0)
|
||||
// With no row axes the sole result row is not an additional rollup row.
|
||||
const number = rows.length === 0 ? '1' : (header.kind === 'data' ? String (++ordinal) : '')
|
||||
if (header.kind !== 'data') labels.push (aggregateLabel (header))
|
||||
return [...(numbered ? [number] : []),
|
||||
...Array.from ({ length: leading }, (_, index) => labels[index] ?? '')]
|
||||
})
|
||||
const spans = rowHeaders.map ((row, index) => rows.map ((_, axis) => {
|
||||
if (row.values.length <= axis) return 1
|
||||
const prefix = JSON.stringify (row.values.slice (0, axis + 1))
|
||||
const previous = rowHeaders[index - 1]
|
||||
if (previous != null && previous.values.length > axis
|
||||
&& JSON.stringify (previous.values.slice (0, axis + 1)) === prefix) return 0
|
||||
let span = 1
|
||||
while (index + span < rowHeaders.length)
|
||||
{
|
||||
const next = rowHeaders[index + span]
|
||||
if (next.values.length <= axis
|
||||
|| JSON.stringify (next.values.slice (0, axis + 1)) !== prefix) break
|
||||
++span
|
||||
}
|
||||
return span
|
||||
}))
|
||||
rowLabels.forEach ((labels, row) => spans[row].forEach ((span, column) => {
|
||||
if (span === 0) labels[column + (numbered ? 1 : 0)] = ''
|
||||
}))
|
||||
// This unmerged matrix is also the single source for TSV and HTML clipboard headings.
|
||||
const valueHeadingRows = Array.from ({ length: levels }, (_, level) => [
|
||||
...(numbered ? [level === 0 ? message ('rowNumberHeading').text : ''] : []),
|
||||
...Array.from ({ length: leading }, (_, index) =>
|
||||
level === 0 && rows[index] != null
|
||||
? countsAxisLabel (rows[index], definition, locale).text : ''),
|
||||
...columnHeaders.map ((header, index) => {
|
||||
if (cols.length === 0) return message ('count').text
|
||||
if (level > header.values.length) return ''
|
||||
if (level === header.values.length)
|
||||
return header.kind === 'data' ? '' : aggregateLabel (header)
|
||||
const prefix = header.values.slice (0, level + 1)
|
||||
const previous = columnHeaders[index - 1]
|
||||
if (level < levels - 1 && previous != null
|
||||
&& JSON.stringify (previous.values.slice (0, level + 1)) === JSON.stringify (prefix))
|
||||
return ''
|
||||
return display (header.values[level], cols[level], rows.length + level)
|
||||
})])
|
||||
const axisHeadingRows = cols.map ((axis, level) => [
|
||||
...(numbered ? [''] : []), ...Array.from ({ length: leading }, () => ''),
|
||||
...columnHeaders.map ((header, index) => {
|
||||
if (header.values.length < level) return ''
|
||||
const previous = columnHeaders[index - 1]
|
||||
const prefix = JSON.stringify (header.values.slice (0, level))
|
||||
return (previous != null && previous.values.length >= level
|
||||
&& JSON.stringify (previous.values.slice (0, level)) === prefix)
|
||||
? '' : countsAxisLabel (axis, definition, locale).text
|
||||
})])
|
||||
const headingRows = cols.length === 0 ? valueHeadingRows
|
||||
: axisHeadingRows.flatMap ((headings, level) => [headings, valueHeadingRows[level]])
|
||||
if (cols.length > 0)
|
||||
{
|
||||
for (let index = 0; index < leading + (numbered ? 1 : 0); ++index)
|
||||
{
|
||||
headingRows[0][index] = valueHeadingRows[0][index]
|
||||
headingRows[1][index] = ''
|
||||
}
|
||||
}
|
||||
const axisHeaderCells = (level: number) => {
|
||||
const output = []
|
||||
for (let index = 0; index < columnHeaders.length; ++index)
|
||||
{
|
||||
const header = columnHeaders[index]
|
||||
if (header.values.length < level) continue
|
||||
const prefix = JSON.stringify (header.values.slice (0, level))
|
||||
let span = 1
|
||||
while (index + span < columnHeaders.length)
|
||||
{
|
||||
const sibling = columnHeaders[index + span]
|
||||
if (sibling.values.length < level
|
||||
|| JSON.stringify (sibling.values.slice (0, level)) !== prefix) break
|
||||
++span
|
||||
}
|
||||
const name = countsAxisLabel (cols[level], definition, locale)
|
||||
output.push (<th key={index} colSpan={span} scope="colgroup"
|
||||
lang={name.language} className="px-2 py-1">{name.text}</th>)
|
||||
index += span - 1
|
||||
}
|
||||
return output
|
||||
}
|
||||
const matrix = rowHeaders.map ((header, index) => [...rowLabels[index],
|
||||
...columnHeaders.map (column => String (count (header, column)))])
|
||||
const headerCells = (level: number) => {
|
||||
const output = []
|
||||
for (let index = 0; index < columnHeaders.length; ++index)
|
||||
{
|
||||
const header = columnHeaders[index]
|
||||
if (cols.length > 0 && level > header.values.length) continue
|
||||
const aggregate = header.kind !== 'data' && level === header.values.length
|
||||
const leaf = level === levels - 1 || aggregate
|
||||
const text =
|
||||
cols.length === 0
|
||||
? message ('count').text
|
||||
: (aggregate ? aggregateLabel (header)
|
||||
: display (header.values[level], cols[level], rows.length + level))
|
||||
let span = 1
|
||||
if (!leaf)
|
||||
{
|
||||
const prefix = JSON.stringify (header.values.slice (0, level + 1))
|
||||
while (index + span < columnHeaders.length)
|
||||
{
|
||||
const sibling = columnHeaders[index + span]
|
||||
if (sibling.values.length <= level
|
||||
|| JSON.stringify (sibling.values.slice (0, level + 1)) !== prefix) break
|
||||
++span
|
||||
}
|
||||
}
|
||||
const sortKey = leaf ? measureKey (header) : `a${rows.length + level}`
|
||||
const field = leaf ? columnKey (header) : sortKey
|
||||
output.push (<CountsSortHeader key={`${level}:${field}:${index}`} field={field}
|
||||
colSpan={span} resizable={leaf}
|
||||
rowSpan={(aggregate && cols.length > 0) ? (levels - level) * 2 - 1 : 1}
|
||||
totalKind={header.kind !== 'data' ? header.kind : undefined}
|
||||
dragId={level === 0 && header.values.length > 0
|
||||
? dragIds[JSON.stringify (header.values.slice (0, 1))] : undefined}
|
||||
label={{ text, language: message ('count').language }}
|
||||
sorts={sorts.map (sort => sort.key === sortKey ? { ...sort, key: field } : sort)}
|
||||
change={(_, additive) => leaf ? changeMeasure (header, additive)
|
||||
: change (sortKey, additive)}/>)
|
||||
index += span - 1
|
||||
}
|
||||
return output
|
||||
}
|
||||
return <div className="min-h-0 flex-1 overflow-auto" data-counts-results aria-busy={pending}>
|
||||
<CountsSelectableTable family={`pivot_${resource}`} columns={keys}
|
||||
copyTable={() => ({ columns: keys.map ((_, index) => ({ label: headingRows[0][index] })),
|
||||
headerRows: headingRows, rows: matrix })}
|
||||
identity={`${countsRequestParams (params)}:${JSON.stringify (keys)}`}
|
||||
className="counts-summary text-left text-sm">
|
||||
<thead className="sticky top-0 z-20">
|
||||
{headingRows.map ((_, tier) => <tr key={tier}>
|
||||
{tier === 0 && <>
|
||||
{numbered && <CountsRowNumber rowSpan={headingLevels}/>}
|
||||
{Array.from ({ length: leading }, (_, index) => rows[index] == null
|
||||
? <th key={index} rowSpan={headingLevels}/>
|
||||
: <CountsSortHeader key={rows[index].field} field={`a${index}`}
|
||||
rowSpan={headingLevels}
|
||||
label={countsAxisLabel (rows[index], definition, locale)}
|
||||
sorts={sorts} change={change}/>)}
|
||||
</>}
|
||||
{(cols.length > 0 && tier % 2 === 0)
|
||||
? axisHeaderCells (tier / 2) : headerCells (Math.floor (tier / 2))}
|
||||
</tr>)}
|
||||
</thead>
|
||||
<tbody>{rowHeaders.map ((row, index) => <tr key={columnKey (row)}>
|
||||
{rowLabels[index].map ((value, offset) => {
|
||||
const axis = offset - (numbered ? 1 : 0)
|
||||
if (axis >= 0 && row.kind !== 'data' && axis > row.values.length) return null
|
||||
const aggregate = axis >= 0 && row.kind !== 'data' && axis === row.values.length
|
||||
const span = axis < 0 ? 1 : spans[index][axis] ?? 1
|
||||
return span === 0 ? null : <td key={offset} rowSpan={span}
|
||||
colSpan={aggregate ? leading - axis : 1}
|
||||
role={axis >= 0 ? 'rowheader' : undefined} data-counts-copy={value}
|
||||
data-counts-aggregate={row.kind !== 'data' ? '' : undefined}
|
||||
data-counts-total={row.kind !== 'data' ? row.kind : undefined}
|
||||
data-counts-row-total={row.kind !== 'data' ? row.kind : undefined}
|
||||
className={cn ('overflow-hidden break-words border px-2 py-1 dark:border-slate-700',
|
||||
numbered && offset === 0 && 'text-right tabular-nums')}>{value}</td>
|
||||
})}
|
||||
{columnHeaders.map (column => <td key={columnKey (column)}
|
||||
data-counts-copy={String (count (row, column))}
|
||||
data-counts-row-total={row.kind !== 'data' ? row.kind : undefined}
|
||||
data-counts-total={(row.kind === 'grand_total' || column.kind === 'grand_total')
|
||||
? 'grand_total' : ((row.kind === 'subtotal' || column.kind === 'subtotal')
|
||||
? 'subtotal' : undefined)}
|
||||
data-counts-aggregate={(row.kind !== 'data' || column.kind !== 'data')
|
||||
? '' : undefined}
|
||||
className="border px-2 py-1 dark:border-slate-700">
|
||||
<PrefetchLink to={drill (row, column)} aria-label={
|
||||
totalLabels.get (JSON.stringify ([row.values, column.values])) == null ? undefined
|
||||
: `${message (totalLabels.get (
|
||||
JSON.stringify ([row.values, column.values]))!).text}: ${count (row, column)}`}>
|
||||
{count (row, column)}</PrefetchLink>
|
||||
</td>)}
|
||||
</tr>)}</tbody>
|
||||
</CountsSelectableTable>
|
||||
</div>
|
||||
}
|
||||
|
||||
|
||||
const CountsSummaryResults = (props: Parameters<typeof CountsSummaryTable>[0]) => {
|
||||
const locale = useLocale ()
|
||||
const { message } = useUITranslation ('counts')
|
||||
const headers = props.data.columnHeaders ?? []
|
||||
const columns = props.data.cols ?? []
|
||||
if (columns.length === 0) return <CountsSummaryTable {...props}/>
|
||||
const groups = [...new Set (headers.filter (header => header.values.length > 0)
|
||||
.map (header => JSON.stringify (header.values.slice (0, 1))))]
|
||||
const ids = groups.map ((_, index) => `group${index}`)
|
||||
const dragIds = Object.fromEntries (groups.map ((group, index) => [group, ids[index]]))
|
||||
return <SortableColumns ids={ids} name={id => ({
|
||||
text: countsSummaryValue (JSON.parse (groups[ids.indexOf (id)])[0], columns[0],
|
||||
props.data.rows?.length ?? 0, props.data, locale,
|
||||
props.params.get ('timezone') ?? 'UTC'), language: message ('count').language })}
|
||||
onChange={order => props.update ('column_order', encodeCountState ('column_order',
|
||||
order.map (id => ({ value: JSON.parse (groups[ids.indexOf (id)])[0] }))))}>
|
||||
{order => {
|
||||
const position = (header: CountsPivotHeader) => header.values.length === 0 ? order.length
|
||||
: order.indexOf (dragIds[JSON.stringify (header.values.slice (0, 1))])
|
||||
const sorted = [...headers].sort ((a, b) => position (a) - position (b))
|
||||
return <CountsSummaryTable {...props} data={{ ...props.data, columnHeaders: sorted }}
|
||||
dragIds={dragIds}/>
|
||||
}}
|
||||
</SortableColumns>
|
||||
}
|
||||
|
||||
export default CountsSummaryResults
|
||||
@@ -0,0 +1,106 @@
|
||||
import { fireEvent, screen } from '@testing-library/react'
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
|
||||
import CountsNestedTable from '@/components/counts/CountsNestedTable'
|
||||
import CountsSelectableTable from '@/components/counts/CountsSelectableTable'
|
||||
import CountsTableLayout from '@/components/counts/CountsTableLayout'
|
||||
import { useCountsTableLayout } from '@/lib/countsTableLayout'
|
||||
import { renderWithProviders } from '@/test/render'
|
||||
|
||||
import type { CountField } from '@/lib/counts'
|
||||
|
||||
|
||||
const renderTable = () => renderWithProviders (<CountsTableLayout>
|
||||
<CountsSelectableTable family="details" columns={['resource_id', 'version_no']}
|
||||
identity="stable" name={{ text: 'Grid', language: 'en' }}>
|
||||
<tbody>
|
||||
<tr><td data-counts-copy="first cell"><span>first</span> <span>cell</span></td><td>second</td></tr>
|
||||
<tr><td>third</td><td><input aria-label="editor" defaultValue="editable"/></td></tr>
|
||||
</tbody>
|
||||
</CountsSelectableTable>
|
||||
</CountsTableLayout>, { route: '/ja/counts/posts' })
|
||||
|
||||
|
||||
describe ('Counts cell keyboard contract', () => {
|
||||
it ('moves Home/End on rows and Ctrl+Home/End on columns with a stable Shift anchor', () => {
|
||||
renderTable ()
|
||||
const table = screen.getByRole ('grid', { name: 'Grid' })
|
||||
table.focus ()
|
||||
fireEvent.keyDown (table, { key: 'End' })
|
||||
expect (screen.getByText ('second')).toHaveAttribute ('data-counts-selected')
|
||||
fireEvent.keyDown (table, { key: 'End', ctrlKey: true, shiftKey: true })
|
||||
expect (screen.getByText ('second')).toHaveAttribute ('data-counts-selected')
|
||||
expect (screen.getByRole ('textbox').closest ('td')).toHaveAttribute ('data-counts-selected')
|
||||
expect (screen.getByText ('third')).not.toHaveAttribute ('data-counts-selected')
|
||||
fireEvent.keyDown (table, { key: 'Home' })
|
||||
expect (screen.getByText ('third')).toHaveAttribute ('data-counts-selected')
|
||||
fireEvent.keyDown (table, { key: 'Home', ctrlKey: true })
|
||||
expect (screen.getByText ('first').closest ('td')).toHaveAttribute ('data-counts-selected')
|
||||
})
|
||||
|
||||
it.each (['Enter', 'Escape']) ('leaves text mode with %s and restores the cell focus', key => {
|
||||
renderTable ()
|
||||
const table = screen.getByRole ('grid', { name: 'Grid' })
|
||||
const cell = screen.getByText ('first').closest ('td')!
|
||||
table.focus ()
|
||||
fireEvent.keyDown (table, { key: 'Home' })
|
||||
fireEvent.keyDown (table, { key: 'Enter' })
|
||||
expect (cell).toHaveAttribute ('data-counts-text')
|
||||
expect (window.getSelection ()?.toString ()).toBe ('first cell')
|
||||
const setData = vi.fn ()
|
||||
fireEvent.copy (table, { clipboardData: { setData } })
|
||||
expect (setData).not.toHaveBeenCalled ()
|
||||
fireEvent.keyDown (table, { key })
|
||||
expect (cell).not.toHaveAttribute ('data-counts-text')
|
||||
expect (cell).toHaveAttribute ('data-counts-selected')
|
||||
expect (window.getSelection ()?.toString ()).toBe ('')
|
||||
expect (table).toHaveFocus ()
|
||||
fireEvent.copy (table, { clipboardData: { setData } })
|
||||
expect (setData).toHaveBeenCalledWith ('text/plain', 'first cell')
|
||||
expect (setData).toHaveBeenCalledWith ('text/html', expect.stringContaining ('first cell'))
|
||||
})
|
||||
|
||||
it ('does not consume input keyboard events and uses the sum of column widths', () => {
|
||||
renderTable ()
|
||||
const input = screen.getByRole ('textbox')
|
||||
input.focus ()
|
||||
expect (fireEvent.keyDown (input, { key: 'Enter' })).toBe (true)
|
||||
expect (input).toHaveFocus ()
|
||||
expect (screen.getByRole ('grid', { name: 'Grid' })).toHaveStyle ({ width: '176px' })
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
const WidthControls = () => {
|
||||
const layout = useCountsTableLayout ()
|
||||
return <>
|
||||
<output>{layout.width ('details', 'title')}</output>
|
||||
<button onClick={() => layout.fit ('details', 'title', 1000)}>fit</button>
|
||||
<button onClick={() => layout.resize ('details', 'title', 180)}>resize</button>
|
||||
</>
|
||||
}
|
||||
|
||||
it ('caps automatic header sizing and preserves manual widths on subsequent fitting', () => {
|
||||
renderWithProviders (<CountsTableLayout><WidthControls/></CountsTableLayout>)
|
||||
fireEvent.click (screen.getByRole ('button', { name: 'fit' }))
|
||||
expect (screen.getByRole ('status')).toHaveTextContent ('640')
|
||||
fireEvent.click (screen.getByRole ('button', { name: 'resize' }))
|
||||
fireEvent.click (screen.getByRole ('button', { name: 'fit' }))
|
||||
expect (screen.getByRole ('status')).toHaveTextContent ('180')
|
||||
})
|
||||
|
||||
it ('wraps nested JSON without truncating, reordering or changing its data', () => {
|
||||
const fields: CountField[] = [
|
||||
{ key: 'id', label: 'counts:columns.resourceId' },
|
||||
{ key: 'name', label: 'counts:columns.name' }]
|
||||
const rows = [{ id: 1, name: `https://example.invalid/${'a'.repeat (300)}` },
|
||||
{ id: 2, name: 'line one\nline two "quoted"' }]
|
||||
const params = new URLSearchParams ({ children: 'tags.id tags.name', nested_json: '1' })
|
||||
const { container } = renderWithProviders (<CountsNestedTable rows={rows} fields={fields}
|
||||
path="tags" resource="posts" title={{ text: 'Tags', language: 'en' }}
|
||||
params={params} update={vi.fn ()}/>, { route: '/ja/counts/posts' })
|
||||
const pre = container.querySelector ('pre')!
|
||||
expect (pre.textContent).toBe (JSON.stringify (rows))
|
||||
expect (pre).toHaveClass ('whitespace-pre-wrap', '[overflow-wrap:anywhere]', 'font-sans')
|
||||
expect (pre).not.toHaveClass ('truncate', 'whitespace-pre', 'select-none')
|
||||
})
|
||||
@@ -0,0 +1,34 @@
|
||||
import { useState } from 'react'
|
||||
|
||||
import { CountsLayoutContext, defaultCountWidths, MAX_COUNT_WIDTH, MIN_COUNT_WIDTH,
|
||||
MAX_INITIAL_COUNT_WIDTH }
|
||||
from '@/lib/countsTableLayout'
|
||||
|
||||
import type { ReactNode } from 'react'
|
||||
|
||||
|
||||
const CountsTableLayout = ({ children }: { children: ReactNode }) => {
|
||||
const [widths, setWidths] = useState<Record<string, number>> ({ })
|
||||
const [headings, setHeadings] = useState<Record<string, number>> ({ })
|
||||
const width = (family: string, key: string) =>
|
||||
widths[`${family}/${key}`]
|
||||
?? Math.max (defaultCountWidths[key] ?? 160, headings[`${family}/${key}`] ?? 0)
|
||||
const fit = (family: string, key: string, value: number) => {
|
||||
const next = Math.ceil (Math.min (MAX_INITIAL_COUNT_WIDTH, value))
|
||||
setHeadings (previous => previous[`${family}/${key}`] === next ? previous
|
||||
: { ...previous, [`${family}/${key}`]: next })
|
||||
}
|
||||
const custom = (family: string, key: string) => widths[`${family}/${key}`] != null
|
||||
const resize = (family: string, key: string, value: number) => {
|
||||
const next = Math.round (Math.max (MIN_COUNT_WIDTH, Math.min (MAX_COUNT_WIDTH, value)))
|
||||
setWidths (previous => previous[`${family}/${key}`] === next ? previous
|
||||
: { ...previous, [`${family}/${key}`]: next })
|
||||
}
|
||||
return (
|
||||
<CountsLayoutContext.Provider value={{ width, custom, resize, fit }}>
|
||||
{children}
|
||||
</CountsLayoutContext.Provider>)
|
||||
}
|
||||
|
||||
|
||||
export default CountsTableLayout
|
||||
@@ -0,0 +1,112 @@
|
||||
.counts-grid > tbody > tr > td {
|
||||
@apply h-6;
|
||||
}
|
||||
|
||||
.counts-summary > tbody > tr > td,
|
||||
.counts-summary > thead > tr > th {
|
||||
padding: 0.5rem 0.75rem;
|
||||
}
|
||||
|
||||
.counts-grid > tbody > tr > td[data-counts-text] {
|
||||
position: relative;
|
||||
outline: 1px solid #64748b;
|
||||
outline-offset: -1px;
|
||||
user-select: text;
|
||||
-webkit-user-select: text;
|
||||
-webkit-touch-callout: default;
|
||||
cursor: text;
|
||||
}
|
||||
|
||||
.dark .counts-grid > tbody > tr > td[data-counts-text] {
|
||||
outline-color: #94a3b8;
|
||||
}
|
||||
|
||||
.counts-text-caret {
|
||||
position: absolute;
|
||||
width: 1px;
|
||||
background-color: currentColor;
|
||||
pointer-events: none;
|
||||
user-select: none;
|
||||
animation: counts-caret-blink 1s step-end infinite;
|
||||
}
|
||||
|
||||
@keyframes counts-caret-blink {
|
||||
0%, 49% { opacity: 1; }
|
||||
50%, 100% { opacity: 0; }
|
||||
}
|
||||
|
||||
@media (prefers-reduced-motion: reduce) {
|
||||
.counts-text-caret { animation: none; }
|
||||
}
|
||||
|
||||
.counts-grid > tbody > tr:nth-child(odd) > td:not([data-counts-selected]) {
|
||||
@apply bg-white text-slate-900;
|
||||
}
|
||||
|
||||
.counts-grid > tbody > tr:nth-child(even) > td:not([data-counts-selected]) {
|
||||
@apply bg-slate-50 text-slate-900;
|
||||
}
|
||||
|
||||
.dark .counts-grid > tbody > tr:nth-child(odd) > td:not([data-counts-selected]) {
|
||||
@apply bg-slate-950 text-slate-100;
|
||||
}
|
||||
|
||||
.dark .counts-grid > tbody > tr:nth-child(even) > td:not([data-counts-selected]) {
|
||||
@apply bg-slate-900 text-slate-100;
|
||||
}
|
||||
|
||||
.counts-grid > thead {
|
||||
@apply bg-slate-200 text-slate-950;
|
||||
}
|
||||
|
||||
.counts-grid > thead > tr > th {
|
||||
@apply border-b border-r border-slate-300 bg-slate-200 text-slate-950;
|
||||
}
|
||||
|
||||
.dark .counts-grid > thead,
|
||||
.dark .counts-grid > thead > tr > th {
|
||||
@apply border-slate-600 bg-slate-800 text-slate-50;
|
||||
}
|
||||
|
||||
.counts-grid > tbody > tr > td {
|
||||
@apply border-r border-slate-200 dark:border-slate-700;
|
||||
-webkit-touch-callout: none;
|
||||
}
|
||||
|
||||
.counts-grid > tbody > tr > td:has(> .counts-nested) {
|
||||
padding: 8px 12px;
|
||||
}
|
||||
|
||||
.counts-nested .counts-grid > thead > tr > th,
|
||||
.counts-nested .counts-grid > tbody > tr > td {
|
||||
padding: 8px 12px;
|
||||
}
|
||||
|
||||
.counts-nested {
|
||||
@apply border-l-2 border-slate-300 dark:border-slate-600;
|
||||
}
|
||||
|
||||
/* Aggregate kind is semantic; only aggregate rows receive a background. */
|
||||
.counts-grid > tbody > tr > td[data-counts-total],
|
||||
.counts-grid > thead > tr > th[data-counts-total] {
|
||||
@apply border-l border-l-slate-400 font-semibold dark:border-l-slate-500;
|
||||
}
|
||||
|
||||
.counts-grid > tbody > tr > td[data-counts-row-total='subtotal']:not([data-counts-selected]) {
|
||||
background-color: #e2e8f0;
|
||||
color: #0f172a;
|
||||
font-weight: 600;
|
||||
}
|
||||
.counts-grid > tbody > tr > td[data-counts-row-total='grand_total']:not([data-counts-selected]) {
|
||||
background-color: #cbd5e1;
|
||||
color: #0f172a;
|
||||
font-weight: 700;
|
||||
}
|
||||
.dark .counts-grid > tbody > tr > td[data-counts-row-total='subtotal']:not([data-counts-selected]) {
|
||||
background-color: #334155;
|
||||
color: #f1f5f9;
|
||||
}
|
||||
.dark .counts-grid > tbody > tr > td[data-counts-row-total='grand_total']:not([data-counts-selected]) {
|
||||
background-color: #475569;
|
||||
color: #fff;
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import { Dialog,
|
||||
DialogHeader,
|
||||
DialogTitle } from '@/components/ui/dialog'
|
||||
import { DialogueContext } from '@/lib/dialogues/useDialogue'
|
||||
import { cn } from '@/lib/utils'
|
||||
import { useUITranslation } from '@/lib/useUITranslation'
|
||||
|
||||
import type { FC, ReactNode } from 'react'
|
||||
@@ -155,6 +156,22 @@ const DialogueProvider: FC<Props> = ({ children }) => {
|
||||
},
|
||||
[closeRequest, pendingIds])
|
||||
|
||||
const dismiss = async (request: DialogueRequest) => {
|
||||
if (request.kind === 'form' && request.options.onCancel != null)
|
||||
{
|
||||
setPendingIds (current => [...current, request.id])
|
||||
try
|
||||
{
|
||||
if (await request.options.onCancel () === false) return
|
||||
}
|
||||
finally
|
||||
{
|
||||
setPendingIds (current => current.filter (id => id !== request.id))
|
||||
}
|
||||
}
|
||||
closeRequest (request.id, request.kind === 'confirm' ? false : null)
|
||||
}
|
||||
|
||||
const active = queue[0]
|
||||
const startActions =
|
||||
active?.kind === 'form'
|
||||
@@ -178,18 +195,17 @@ const DialogueProvider: FC<Props> = ({ children }) => {
|
||||
nestedConfirm?.parentId === active.id
|
||||
|| pendingIds.includes (active.id)
|
||||
if (!(open) && !(blocked))
|
||||
closeRequest (active.id, active.kind !== 'confirm' && null)
|
||||
dismiss (active)
|
||||
}}>
|
||||
<DialogContent
|
||||
closeLabel={(active.kind === 'confirm' || active.kind === 'choice')
|
||||
? active.options.closeLabel : undefined}
|
||||
className={
|
||||
active.kind === 'form'
|
||||
? `flex max-h-[calc(100dvh-1rem)] ${ active.options.size === 'large'
|
||||
? 'max-w-3xl'
|
||||
: 'max-w-lg' } flex-col overflow-hidden gap-0`
|
||||
: 'px-6 pb-6 pt-7'
|
||||
}
|
||||
className={cn (active.kind === 'form'
|
||||
? 'flex max-h-[calc(100dvh-1rem)] flex-col overflow-hidden gap-0'
|
||||
: 'px-6 pb-6 pt-7', active.kind === 'form' && (
|
||||
active.options.size === 'viewport'
|
||||
? 'h-[88dvh] w-[88vw] max-w-none'
|
||||
: (active.options.size === 'large' ? 'max-w-3xl' : 'max-w-lg')))}
|
||||
onEscapeKeyDown={event => {
|
||||
if (nestedConfirm?.parentId === active.id || pendingIds.includes (active.id))
|
||||
event.preventDefault ()
|
||||
@@ -237,11 +253,14 @@ const DialogueProvider: FC<Props> = ({ children }) => {
|
||||
</Button>))}
|
||||
</div>
|
||||
|
||||
<div className="flex w-full flex-col gap-2 md:w-auto md:flex-row">
|
||||
<div className={cn (
|
||||
'flex w-full gap-2 md:w-auto md:flex-row',
|
||||
active.options.footerLayout === 'inline'
|
||||
? 'flex-row justify-end [&>button]:w-auto' : 'flex-col')}>
|
||||
<Button
|
||||
className="w-full md:w-auto"
|
||||
variant="outline"
|
||||
onClick={() => closeRequest (active.id)}
|
||||
onClick={() => dismiss (active)}
|
||||
disabled={pendingIds.includes (active.id)
|
||||
|| nestedConfirm?.parentId === active.id}>
|
||||
{active.options.cancelText ?? label ('actions.cancel')}
|
||||
|
||||
@@ -8,10 +8,11 @@ import type { FC, ReactNode } from 'react'
|
||||
|
||||
type Props = {
|
||||
children: ReactNode
|
||||
className?: string }
|
||||
className?: string
|
||||
animateLayout?: boolean }
|
||||
|
||||
|
||||
const MainArea: FC<Props> = ({ children, className }) => {
|
||||
const MainArea: FC<Props> = ({ children, className, animateLayout = true }) => {
|
||||
const behaviourSettings = useClientBehaviourSettings ()
|
||||
const animationMode = behaviourSettings.animation ?? 'normal'
|
||||
const layoutTransition = clientAnimationTransition (
|
||||
@@ -23,7 +24,7 @@ const MainArea: FC<Props> = ({ children, className }) => {
|
||||
<motion.main
|
||||
transition={{ layout: layoutTransition }}
|
||||
className={cn ('flex-1 overflow-y-auto p-4', className)}
|
||||
layout={animationMode === 'off' ? false : 'position'}>
|
||||
layout={(animationMode === 'off' || !animateLayout) ? false : 'position'}>
|
||||
{children}
|
||||
</motion.main>)
|
||||
}
|
||||
|
||||
@@ -15,6 +15,8 @@ type Props = {
|
||||
children: ReactNode
|
||||
className?: string
|
||||
maxWidth?: number
|
||||
maxWidthRatio?: number
|
||||
animateLayout?: boolean
|
||||
onWidthChange?: (width: number) => void
|
||||
sidebarKey: string
|
||||
side?: SidebarSide }
|
||||
@@ -67,6 +69,8 @@ const SidebarComponent: FC<Props> = ({
|
||||
children,
|
||||
className,
|
||||
maxWidth,
|
||||
maxWidthRatio,
|
||||
animateLayout = true,
|
||||
onWidthChange,
|
||||
sidebarKey,
|
||||
side = 'left',
|
||||
@@ -80,6 +84,7 @@ const SidebarComponent: FC<Props> = ({
|
||||
)
|
||||
const rootRef = useRef<HTMLDivElement | null> (null)
|
||||
const maxWidthRef = useRef (maxWidth)
|
||||
const maxWidthRatioRef = useRef (maxWidthRatio)
|
||||
const getContainerWidth = useCallback (() => {
|
||||
const containerWidth = rootRef.current
|
||||
?.closest ('[data-sidebar-container]')
|
||||
@@ -93,12 +98,14 @@ const SidebarComponent: FC<Props> = ({
|
||||
const viewportWidth =
|
||||
typeof window === 'undefined' ? containerWidth : window.innerWidth
|
||||
const viewportMaxWidth = Math.floor (viewportWidth * MAX_VIEWPORT_WIDTH_RATIO)
|
||||
const maxAllowedWidth =
|
||||
maxWidthRef.current == null
|
||||
const ratioMaxWidth = maxWidthRatioRef.current == null
|
||||
? Infinity : Math.floor (containerWidth * maxWidthRatioRef.current)
|
||||
const configuredMaxWidth = maxWidthRef.current == null
|
||||
? getSidebarMaxWidth (containerWidth)
|
||||
: Math.max (
|
||||
MIN_SIDEBAR_WIDTH,
|
||||
: Math.max (MIN_SIDEBAR_WIDTH,
|
||||
Math.min (viewportMaxWidth, maxWidthRef.current))
|
||||
const maxAllowedWidth = Math.max (MIN_SIDEBAR_WIDTH,
|
||||
Math.min (ratioMaxWidth, configuredMaxWidth))
|
||||
|
||||
return Math.min (maxAllowedWidth, Math.max (MIN_SIDEBAR_WIDTH, nextWidth))
|
||||
}, [])
|
||||
@@ -171,10 +178,11 @@ const SidebarComponent: FC<Props> = ({
|
||||
|
||||
useEffect (() => {
|
||||
maxWidthRef.current = maxWidth
|
||||
maxWidthRatioRef.current = maxWidthRatio
|
||||
const containerWidth = getContainerWidth ()
|
||||
const nextWidth = getClampedWidth (widthRef.current, containerWidth)
|
||||
applyWidth (nextWidth)
|
||||
}, [applyWidth, getClampedWidth, getContainerWidth, maxWidth])
|
||||
}, [applyWidth, getClampedWidth, getContainerWidth, maxWidth, maxWidthRatio])
|
||||
|
||||
const updateWidth = useCallback ((clientX: number) => {
|
||||
const state = resizeRef.current
|
||||
@@ -310,7 +318,7 @@ const SidebarComponent: FC<Props> = ({
|
||||
return (
|
||||
<motion.div
|
||||
ref={rootRef}
|
||||
layout={animationMode === 'off' ? false : 'position'}
|
||||
layout={(animationMode === 'off' || !animateLayout) ? false : 'position'}
|
||||
transition={{ layout: layoutTransition }}
|
||||
style={style}
|
||||
className={cn (
|
||||
|
||||
@@ -7,6 +7,7 @@ import { useTranslation } from 'react-i18next'
|
||||
|
||||
import LocalisedText from '@/components/LocalisedText'
|
||||
import { translateUIMessage } from '@/lib/i18n'
|
||||
import { useClientMotion } from '@/lib/useClientMotion'
|
||||
import { cn } from '@/lib/utils'
|
||||
|
||||
const Dialog = DialogPrimitive.Root
|
||||
@@ -37,14 +38,17 @@ const DialogContent = React.forwardRef<
|
||||
React.ComponentPropsWithoutRef<typeof DialogPrimitive.Content>
|
||||
& { closeLabel?: React.ReactNode }
|
||||
>(
|
||||
({ className, children, closeLabel, ...props }, ref) => {
|
||||
({ className, children, closeLabel, style, ...props }, ref) => {
|
||||
const { i18n } = useTranslation ('shared')
|
||||
const motion = useClientMotion ()
|
||||
const animationDuration = `${motion.duration ('popover')}ms`
|
||||
const close = translateUIMessage ('dialogue.close', 'shared', i18n.language)
|
||||
return (
|
||||
<DialogPortal>
|
||||
<DialogOverlay/>
|
||||
<DialogOverlay style={{ animationDuration }}/>
|
||||
<DialogPrimitive.Content
|
||||
ref={ref}
|
||||
style={{ animationDuration, ...style }}
|
||||
className={cn (
|
||||
'fixed left-[50%] top-[50%] z-50 grid w-[calc(100%-2rem)] max-w-lg',
|
||||
'translate-x-[-50%] translate-y-[-50%]',
|
||||
|
||||
@@ -20,6 +20,7 @@ import type { RenderContext } from '@/lib/renderContext'
|
||||
import type { User } from '@/types'
|
||||
|
||||
export { renderSitemap } from '@/lib/sitemap.server'
|
||||
export { isCountsPath } from '@/lib/routeLocales'
|
||||
|
||||
type RenderedPage = {
|
||||
status: number
|
||||
@@ -66,7 +67,9 @@ export const render = async (
|
||||
authentication: Pick<RenderContext, 'cookie' | 'forwardedFor'> = { },
|
||||
): Promise<RenderedPage | RouteResponse> => {
|
||||
const url = new URL (address, origin)
|
||||
const legacy = legacyLocalePath ({ pathname: url.pathname, search: url.search, hash: '' })
|
||||
const legacy = legacyLocalePath (
|
||||
{ pathname: url.pathname, search: url.search, hash: url.hash },
|
||||
pathname => resolvePageRoute (pathname) != null)
|
||||
const pathname = legacy == null ? url.pathname : new URL (legacy, origin).pathname
|
||||
const segment = pathname.split ('/')[1]
|
||||
const known = resolvePageRoute (pathname)
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
@import "@fontsource-variable/noto-sans-jp";
|
||||
@import './components/counts/countsTable.css';
|
||||
|
||||
@tailwind base;
|
||||
@tailwind components;
|
||||
|
||||
@@ -199,3 +199,43 @@ export const apiDelete = async <T = void> (
|
||||
|
||||
export const isApiError = <T = unknown> (err: unknown): err is AxiosError<T> =>
|
||||
axios.isAxiosError (err)
|
||||
|
||||
|
||||
// The server acknowledges a complete file before native download starts.
|
||||
export const apiDownload = async (
|
||||
path: string,
|
||||
body: Record<string, string>,
|
||||
locale: LocaleCode,
|
||||
onError?: () => void,
|
||||
): Promise<void> => {
|
||||
const { token } = await apiPost<{ token: string }> (`${path}/prepare`, body, { locale })
|
||||
if (typeof token !== 'string' || token.length === 0)
|
||||
throw new Error ('Missing download token')
|
||||
startAPIDownload (path, token, onError)
|
||||
}
|
||||
|
||||
|
||||
export const startAPIDownload = (
|
||||
path: string,
|
||||
token: string,
|
||||
onError?: () => void,
|
||||
query?: URLSearchParams,
|
||||
): (() => void) => {
|
||||
const frame = document.createElement ('iframe')
|
||||
frame.hidden = true
|
||||
frame.addEventListener ('load', () => {
|
||||
const document = frame.contentDocument
|
||||
if (document != null && document.URL !== 'about:blank')
|
||||
{
|
||||
onError?.()
|
||||
frame.remove ()
|
||||
}
|
||||
})
|
||||
frame.addEventListener ('error', () => { onError?.(); frame.remove () })
|
||||
const suffix = query == null ? '' : `?${query.toString ()}`
|
||||
frame.src = `${API_BASE_URL}${path}/files/${encodeURIComponent (token)}${suffix}`
|
||||
document.body.append (frame)
|
||||
// Attachments do not reliably fire load; keep this isolated from page navigation.
|
||||
const timer = window.setTimeout (() => frame.remove (), 10 * 60_000)
|
||||
return () => { window.clearTimeout (timer); frame.remove () }
|
||||
}
|
||||
|
||||
@@ -32,3 +32,9 @@ export const clientScrollBehaviour = (
|
||||
animationMode: ClientAnimationMode,
|
||||
): ScrollBehavior =>
|
||||
animationMode === 'off' ? 'auto' : 'smooth'
|
||||
|
||||
export const clientMotionTokens = {
|
||||
durationMS: { feedback: 120, popover: 160, collapse: 220, reorder: 200, drop: 180 },
|
||||
easing: [.2, 0, 0, 1] as const,
|
||||
cssEasing: 'cubic-bezier(.2, 0, 0, 1)',
|
||||
distance: 4 }
|
||||
|
||||
変更されたファイルが多すぎるため、一部のファイルは表示されません さらに表示
新しいイシューから参照
ユーザーをブロックする